Continuous spectrum monitoring

23.12.2014 19:40

Just before the holidays and with some help from my colleagues at the Institute I managed to finally deploy the first spectrum sensor based on my new UHF receiver design. A handful of similar deployments are planned for the next year. At the end of January I'm traveling to London to mount two similar sensors that will monitor the TV White Spaces pilot in UK as part of a cooperation between King's College London and the CREW project I'm working for.

VESNA stand-alone spectrum sensor in a metal box.

The sensor is based on the VESNA sensor node and is mounted in a small, weather-proof metal box. The previous generation of spectrum sensors (like those that were deployed in our testbed in Logatec) were designed to work in a wireless sensor network. However the low bit rate and reliability of the sensor network and interference caused by another radio next to a sensitive receiver proved to be very limiting. So for this device I went with wired Ethernet even though that reduces the deployment possibilities somewhat. Ethernet now provides a sufficient bit rate to allow for continuous monitoring of the complete UHF band with increased precision that is possible with the new receiver.

In some modes of operation however, the bandwidth from the sensor is still a limiting factor. The Digi Connect ME Ethernet module used here only allows for streaming data up to around 230 kbit/s, which is around 2-3 times slower than what the VESNA's CPU can handle. This is not a problem if you do signal processing on-board the sensor and only send some test statistics back over the network. In that case other things, like the local oscillator and AGC settle time, are usually the limiting factor. However if you want to stream raw signal samples, the network becomes the bottleneck.

Cluster of antennas mounted on JSI building C.

Antenna for the sensor is mounted on top of one of the buildings at the Jožef Stefan Institute campus in Ljubljana. Most of the setup, like power and Ethernet connection, is shared with WLAN Slovenija, who already had some equipment mounted there. On the picture above, my UHF antenna (Super Scan Stick from Moonraker) is second from the left (one with three radials). I'm still using SO-239 connectors and RG-58 coax which I know is not optimal, but they were what I had in stock.

There is direct line of sight to a DVB-T receiver near Ljubljana Castle around 1500 m away and two cellular towers on nearby buildings, so there are plenty of strong transmissions to look at. On the same mast there is also a big omni-directional antenna for 2.4 GHz Wi-Fi. This might cause some interference with sensing in the 800 MHz band (2.4 GHz is the third harmonic of a 800 MHz LO signal), but I don't think that will be a problem. RG-58 cable is bad enough even at sub-1 GHz frequencies and that part of the spectrum seems to be occupied by a strong LTE signal anyway.

Screenshot of the real-time spectrogram plot.

If you click on the image above, you should see a live visualization in you browser of the data stream coming from the sensor.

At the time of writing, the receiver is sweeping the band between 470 MHz and 861 MHz in 1 MHz increments. For each 1 MHz channel, it takes 25000 samples of the baseband signal. From these samples, it calculates the baseband signal power and a vector of sample covariances. It then sends these statistics back to a server that logs them to a hard drive and also provides the visualization (using a somewhat convoluted setup).

The signal power is shown on the spectrogram plot in logarithmic scale. Currently on the spectrogram you can see four DVB-T multiplexes at 482, 522, 562 and 610 MHz and some LTE activity around 800 MHz. Note that the power level is in relative units: 0 dB is the maximum ADC level which can correspond to different absolute power levels due to automatic changes in gain. The sensor can provide calibrated absolute power figures as well, but they are not shown at the moment.

Sample covariances are currently used by a maximum auto-correlation detector to detect whether a channel is vacant or not. This is shown on the live graph below the spectrogram. Note though that this detector works best for narrow-band transmissions, like wireless microphones, and often marks a channel vacant when it is in fact occupied by a wide-band transmission like DVB-T (see my seminar on spectrum sensing methods for details and references).

Posted by Tomaž | Categories: Life | Comments »

Arts & Crafts

19.12.2014 22:44

I have in front of me two sketchbooks. Between them, they contain around a hundred pages and a bundle of loose sheets. Thumbing through them, they start with embarrassingly clumsy pencil sketches, half way through change to line marker drawings and end with digitally shaded print-outs. The first page is dated April 2013. A year and a half later, I'm trying to come up with a reasonable story behind all of this.

The fact is that free-hand drawing is one skill I never really learned. I am pretty handy with a pencil, mind you. I am old enough to have had technical drawing lessons in school that did not involve anything fancier than a compass and a straightedge. I do majority of my notes in a paper notebook and still prefer to do plans and schematics for my home projects with a pen instead of a mouse. However the way one draws objects that do not consist of right angles and straight lines has always eluded me.

Drawing is really hard.

-- Randal Munroe in an recent interview

I have grown up with characters drawn by Miki Muster and I have kept my appetite for comics ever since. I don't think I ever attempted to draw one though. I remember the art class I attended in elementary school. It seems the local artist that held it considered it his job to protect our unspoiled childhood creativity. Any art inspired by popular culture was frowned upon and a reason for a lecture on how completely devoid of imagination our generation was. I guess we stuck to whatever kind of imaginary things children were supposed to draw back then and I was more interested in the room full of ZX Spectrums next door anyway.

So from that point of view, attempting to draw cartoon characters has been an interesting new challenge, something very different from my usual pursuits. Maybe it shouldn't be surprising that I picked up something like that now. I have been writing software at home while studying electronics at the Faculty and I have been doing electronics at home while writing software at work. These days I spend most of my time doing everything from electronics design, software development to writing and giving talks for the Institute. It leaves little space to decompress after work. Doing some non-digital creativity is refreshing.

A working pony, a pony of science.

The other part of this story is, of course, that I've been spending increasing amounts of time on forums and events that have something to do with cartoonish horses and other such things. For better or for worse, it's been a common pass time and means of escape for me this past year. I certainly wouldn't go out and buy a sketch book and a set of pencils if I wouldn't witness a few people from science and engineering professions try themselves in drawing, sometimes with really nice results. It seems that whatever community I happen to meet, I can't manage to stay just a passive observer for long.

If nothing else, that teacher from elementary school was probably right regarding lack of imagination. I don't pretend that what I've made is very original nor that I've managed to teach myself anything else than basics. On the other hand, learning anything necessarily involves some degree of copying. I don't feel this has been a waste even if there is a crowd on the Internet is doing similar things.

The point is that to be truly adept at designing something, you have to understand how it works. [Otherwise] you're drawing ponies. Don't draw ponies.

-- David Kadavy in Design for Hackers

Several years and two schools after that art class I mentioned, I found myself at an art history lecture. It was taught by a bitter old lady who, despite her quite obvious disappointment at the world, was still very much determined to teach us ignorant youth. In addition to explaining historic European architecture styles to us, she also commonly managed to slip in her observations about life in general. On one such occasion she told us how often men fall into the trap of irrationally obsessing over one thing. I remember duly noting it in my notebook.

Maybe this is what she had in mind, maybe not. The fact is that I enjoyed doing it and learning to draw made me look at things around me in new ways. If someone else liked the few results that I shared on the Internet, that much better. Some say it might look terribly silly a few years on, but I'm sure I won't regret gaining a new skill.

Posted by Tomaž | Categories: Life | Comments »

Python's disappearing character trick

08.12.2014 19:42

While working on some improvements to the way Unidecode behaves on "narrow" Unicode Python builds, I stumbled upon an interesting bug in Python 2.7. I noticed it because some unit tests passed only once after saving the .py file, but always failed afterwards. Since the unit tests did not save any state, that behavior appeared very suspicious.

I managed to distill the cause to the following example:

$ python -V
Python 2.7.3
$ cat a.py
from b import foo
foo()
$ cat b.py
def foo():
	s_sp_1 = u'\ud835\udce3'
	print len(s_sp_1)
$ python a.py
2
$ python a.py
1
$ python a.py
1

Notice how the program prints out "2" on the first invocation but on "1" all subsequent calls. You can make it print 2 again by deleting b.pyc.

What is happening here? The Unicode string u'\ud835\udce3' actually contains only one character. However, this character is written as a pair of surrogate characters. The string is equivalent to u'\U0001d4e3'.

Surrogate characters are a hack in Unicode that allows implementations that only support 16-bit code points to handle more than 65536 characters. Basically, it introduces variable-length encoding with 16-bit code units (similarly to how UTF-8 encoding encodes a character in a variable number of 8-bit code units). The problem is that for a while, systems with 16-bit code points were considered to always have one code unit per code point. To muddle up things further, even systems that have 32-bit code units (like most Python builds these days) are supposed to support surrogate pair encoding. This leads to yet another case where very different sequences of code points can produce equivalent Unicode strings (there are quite a few others as well)

What appears to happen in the code above is that Python re-encodes the Unicode string when writing out the compiled version of the b.py module to b.pyc. When re-encoding, it doesn't use the surrogate pair (which makes sense, since my Python build can handle 32-bit code points just fine). However, that means that when reading the b.pyc file on the second invocation, the length of the string is different.

This is actually a bug in Python that has been known since 2008. It's fixed in Python 3.2 and above, but seems to remain unfixed in all Python 2 versions I tried. I tend to agree with Python developers though. It's unusual to have this kind of strings written directly in Python source code and even more unusual to notice the change in encoding between runs. However I might be influenced by the fact that I didn't waste more than a couple of minutes because of this bug.

Posted by Tomaž | Categories: Code | Comments »