Materialized Munin display

15.05.2016 21:25

Speaking of Munin, here's a thing that I've made recently: A small stand-alone display that cycles through a set of measurements from a Munin installation.

Munin display

(Click to watch Munin display video)

Back when ESP8266 chip was the big new thing I ordered a bag of them from eBay. The said bag then proceeded to gather dust in the corner of my desk for a year or so, as such things unfortunately tend to do these days. I also had a really nice white transflective display left over from another project (suffice to say, it cost around 20 £ compared to ones you can get for a tenth of the price with free shipping on DealExtreme). So something like this looked like a natural thing to make.

The hardware is not worth wasting too many words on: an ESP8266 module handles radio and the networking part. The display is a 2-line LCD panel using the common 16-pin interface. An Arduino Pro Mini acts as glue between the display and the ESP8266. There are also 3.3 V (for ESP8266) and 5 V (for LCD and Arduino) power supplies and a transistor level shifter for the serial line between ESP8266 and the Arduino.

ESP8266 runs stock firmware that exposes a modem-like AT-command interface on a serial line. I could have omitted the Arduino and ran the whole thing from the ESP8266 alone, however the lack of GPIO lines on the module I was using meant that I would have to use some kind of GPIO extender or multiplexer to run the 16-pin LCD interface. Arduino with the WeeESP8266 library just seemed less of a hassle.

Top side of the circuit in the Munin display.

From the software side, the device basically acts as a dumb display. The ESP8266 listens on a TCP socket and Arduino pushes everything that is received on that socket to the LCD. All the complexity is hidden in a Python daemon that runs on my CubieTruck. The daemon uses PyMunin to periodically query Munin nodes, renders the strings to be displayed and sends them to the display.

Speaking of ESP8266, my main complaint would be that there is basically zero official documentation about it. Just getting it to boot means reconciling conflicting information from different blog and forum posts (for me, both CH_PD and RST/GPIO16 needed to be pulled low). No one mentioned that RX pin has an internal pull-up. I also way underestimated the current consumption (it says 1 mA stand-by on the datasheet after all and the radio is mostly doing nothing in my case). It turns out that a linear regulator is out of the question and a 3.3 V switch-mode power supply is a must.

My module came with firmware that was very unreliable. Getting official firmware updates from a sticky forum post felt kind of shady and it took some time to get an image that worked with 512 kB flash on my module. That said, the module has been working without resets or hangs for a couple of weeks now which is nice and not something that all similar radio modules are capable of.

Inside the Munin display.

Finally, this is also my first 3D printed project and I learned several important lessons. It's better to leave too much clearance than too little between parts that are supposed to fit together. This box took about four hours of careful sanding and cutting before the top part could be inserted into the bottom since the 3D printer randomly decided to make some walls 1 mm thicker than planned. Also, self-tapping screws and automagically hollowed-out plastic parts don't play nice together.

With all the careful measuring and planning required to come up with a CAD drawing, I'm not sure 3D printing saved me any time compared to a simple plywood box which I could make and fit on the fly. Also, relying on the flexibility and precision of a 3D print made me kind of forget about the mechanical design of the circuit. I'm not particularly proud of the way things fit together and how it looks inside, but most of it is hidden away from view anyway and I guess it works well enough for a quick one-off project.

Posted by Tomaž | Categories: Life | Comments »

Clockwork, part 2

10.04.2016 19:39

I hate to leave a good puzzle unsolved. Last week I was writing about a cheap quartz mechanism I got from an old clock that stopped working. I said that I could not figure out why its rotor only turns in one direction given a seemingly symmetrical construction of the coil that drives it.

There is quite a number of tear downs and descriptions of how such mechanisms work on the web. However, very few seem to address this issue of direction of rotation and those that do don't give a very convincing argument. Some mention that the direction has something to do with the asymmetric shape of the coil's core. This forum post mentions that the direction can be reversed if a different pulse width is used.

So, first of all I had a closer look at the core. It's made of three identical iron sheets, each 0.4 mm thick. Here is one of them on the scanner with the coil and the rotor locations drawn over it:

Coil location and direction of rotation.

It turns out there is in fact a slight asymmetry. The edges of the cut-out for the rotor are 0.4 mm closer together on one diagonal than on the other. It's hard to make that out with unaided eye. It's possible that the curved edge on the other side makes it less error prone to construct the core with all three sheets in same orientation.

Dimension drawing of the magnetic core.

The forum post about pulse lengths and my initial thought about shaded pole motors made me think that there is some subtle transient effect in play that would make the rotor prefer one direction over the other. Using just a single coil, core asymmetry cannot result in a rotating magnetic field if you assume linear conditions (e.g. no part of the core gets saturated) and no delay due to eddy currents. Shaded pole motors overcome this by delaying magnetization of one part of the core through a shorted auxiliary winding, but no such arrangement is present here.

I did some measurements and back-of-the-envelope calculations. The coil has approximately 5000 turns and resistance of 215 Ω. The field strength is nowhere near saturation for iron. The current through the coil settles somewhere on the range of milliseconds (I measured a time constant of 250 μs without the core in place). It seems unlikely any transients in magnetization can affect the movements of the rotor.

After a bit more research, I found out that this type of a motor is called a Lavet type stepping motor. In fact, its operation can be explained completely using static fields and transients don't play any significant role. The rotor has four stable points: two when the coil drives the rotor in one or the other direction and two when the rotor's own permanent magnetization attracts it to the ferromagnetic core. The core asymmetry creates a slight offset between the former and the latter two points. Wikipedia describes the principle quite nicely.

To test this principle, I connected the coil to an Arduino and slowly stepped this clockwork motor through it's four states. The LED on the Arduino board above shows when the coil is energized. The black dot on the rotor roughly marks the position of one of its poles. You can see that when the coil turns off, the rotor turns slightly forward as its permanent magnet aligns it with the diagonal on the core that has a smaller air gap (one step is a bit more pronounced than the other on the video above). This slight forward advancement from the neutral position then makes the rotor prefer the forward over the backward motion when the coil is energized in the other direction.

It's always fascinating to see how a mundane thing like a clock still manages to have parts in it whose principle of operation is very much not obvious from the first glance.

Posted by Tomaž | Categories: Life | Comments »

Clockwork

28.03.2016 15:17

Recently one of the clocks in my apartment stopped. It's been here since before I moved in and is probably more than 10 years old. The housing more or less crumbled away as I opened it. On the other hand the movement inside looked like it was still in a good condition, so I had a look if there was anything in it that I could fix.

Back side of the quartz clock movement.

This is a standard 56 mm quartz wall clock movement. It's pretty much the same as in any other cheap clock I've seen. In this case, its makers were quick to dispel any myths about its quality: no jewels in watchmaker's parlance means no quality bearings and I'm guessing unadjusted means that the frequency of its quartz oscillator can't be adjusted.

Circuit board in the clock movement.

As far as electronics is concerned, there's not much to see in there. There's a single integrated circuit, bonded to a tiny PCB and covered with a blob of epoxy. It uses a small tuning-fork quartz resonator to keep time. As the cover promised, there's no sign of a trimmer for adjusting the quartz load capacitance. Two exposed pads on the top press against some metallic strips that connect to the single AA battery. The life time of the battery was probably more than a year since I don't remember the last time I had to change it.

Coil from the clock movement.

The circuit is connected to a coil on the other side of the circuit board. It drives the coil with 30 ms pulses once per second with alternating polarity. The oscilloscope screenshot below shows voltage on the coil terminals.

Voltage waveform on the two coil terminals.

When the mechanism is assembled, there's a small toroidal permanent magnet sitting in the gap in the coil's core with the first plastic gear on top of it. The toroid is laterally magnetized and works as a rotor in a simple stepper motor.

Permanent magnet used as a rotor in clock movement.

The rotor turns half a turn every second and this is what gives off the audible tick-tock sound. I'm a bit puzzled as to what makes it turn only in one direction. I could see nothing that would work as a shaded pole or something like that. The core also looks perfectly symmetrical with no features that would make it prefer one direction of rotation over the other. Maybe the unusual cutouts on the gear for the second hand have something to do with it.

Update: my follow-up post explains what determines direction of rotation.

Top side of movement with gears in place.

This is what the mechanism looks like with gears in place. The whole construction is very finicky and a monument to material cost reduction. There's no way to run it without the cover in place since gears fall over and the impulses in the coil actually eject the rotor if there's nothing on top holding it in place (it's definitely not as well behaved as one in this video). In fact, I see no traces that the rotor magnet has been permanently bonded in any way with the first gear. It seems to just kind of jump around in the magnetic field and drive the mechanism by rubbing against the inside of the gear.

In the end, I couldn't find anything obviously wrong with this thing. The electronics seem to work correctly. The gears also look and turn fine. When I put it back together it would sometimes run, sometimes it would just jump one step back and forth and sometimes it would stand still. Maybe some part wore down mechanically, increasing friction. Or maybe the magnet lost some of its magnetization and no longer produces enough torque to reliably turn the mechanism. In any case, it's going into the scrap box.

Posted by Tomaž | Categories: Life | Comments »

The problem with gmail.co

14.03.2016 19:41

At this moment, the gmail.co (note missing m) top-level domain is registered by Google. This is not surprising. It's common practice these days for owners of popular internet services to buy up domains that are similar to their own. It might be to fight phising attacks (e.g. go-to-this-totally-legit-gmail.co-login-form type affairs), prevent typosquatting or purely for convenience to redirect users that mistyped the URL to the correct address.

$ whois gmail.co
(...)
Registrant Organization:                     Google Inc.
Registrant City:                             Mountain View
Registrant State/Province:                   CA
Registrant Country:                          United States

gmail.co currently serves a plain 404 Not Found page on the HTTP port. Not really user friendly, but I guess it's good enough to prevent web-based phising attacks.

Now, with half of the world using ...@gmail.com email addresses, it's not uncommon to also mistakenly send an email to a ...@gmail.co address. Normally, if you mistype the domain part of the email address, your MTA will see the DNS resolve fail and you would immediately get either a SMTP error at the time of submission, or a bounced mail shortly after.

Unfortunately, gmail.co domain actually exists, which means that MTAs will in fact attempt to deliver mail to it. There's no MX DNS record, however SMTP specifies that MTAs must in that case use the address in A or AAAA records for delivery. Those do exist (as they allow the previously mentioned HTTP error page to be served to a browser).

To further complicate the situation, the SMTP port 25 on IPs referenced by those A and AAAA records is blackholed. This means that a MTA will attempt to connect to it, hang while the remote host eats up SYN packets, and fail after the TCP handshake timeouts. A timeout looks to the MTA like an unresponsive mail server, which means it will continue to retry the delivery for a considerable amount of time. The RFC 5321 says that it should take at least 4-5 days before it gives up and sends a bounce:

Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days. It MAY be appropriate to set a shorter maximum number of retries for non- delivery notifications and equivalent error messages than for standard messages. The parameters to the retry algorithm MUST be configurable.

In a nutshell, what all of this means is that if you make a typo and send a mail to @gmail.co, it will take around a week for you to receive any indication that your mail was not delivered. Needless to say, this is bad. Especially if the message you were sending was time critical in nature.

Update: Exim will warn you when a message has been delayed for more than 24 hours, so you'll likely notice this error before the default 6 day retry timeout. Still, it's annoying and not all MTAs are that friendly.

The lesson here is that, if you register your own typosquatting domains, do make sure that mail sent to them will be immediately bounced. One way is to simply set an invalid MX record (this is an immediate error for SMTP). You can also run a SMTP server that actively rejects all incoming mail (possibly with a friendly error message reminding the user of the mistyped address), but that requires some more effort.

As for this particular Google's blunder, a workaround is to put a special retry rule for gmail.co in your MTA so that it gives up faster (e.g. see Exim's Retry configuration).

Posted by Tomaž | Categories: Life | Comments »

Some SPF statistics

21.02.2016 19:04

Some people tend their backyard gardens. I host my own mail server. Recently, there has been a push towards more stringent mail server authentication to fight spam and abuse. One of the simple ways of controlling which server is allowed to send mail for a domain is the Sender Policy Framework. Zakir Durumeric explained it nicely in his Neither Snow Nor Rain Nor MITM talk at the 32C3.

The effort for more authentication seems to be headed by Google. That is not surprising. Google Mail is e-mail for most people nowadays. If anyone can push for changes in the infrastructure, it's them. A while ago Google published some statistics regarding the adoption of different standards for their inbound mail. Just recently, they also added visible warnings for their users if mail they received has not been sent from an authenticated server. Just how much an average user can do about that (except perhaps pressure their correspondents to start using Google Mail) seems questionable though.

Anyway, I implemented a SPF check for inbound mail on my server some time ago. I never explicitly rejected mail based on it however. My MTA just adds a header to incoming messages. I was guessing the added header may be picked out by the Bayesian spam filter, if it became significant at any point. After reading about Google's efforts I was wondering what the situation regarding SPF checks looks like for me. Obviously, I see a very different sample of the world's e-mail traffic as Google's servers.

For this experiment I took a 3 month sample of inbound e-mail that was received by my server between November 2015 and January 2016. The mail was classified by Bogofilter into spam and non-spam mail, mostly based on textual content. SPF records were evaluated by spf-tools-perl upon reception. Explanation of results (what softfail, permerror, etc. means) is here.

SPF evaluation results for non-spam messages.

SPF evaluation results for spam messages.

As you can see, the situation in this little corner of the Internet is much less optimistic than the 95.3% SPF adoption rate that Google sees. More than half of mail I see doesn't have a SPF record. A successful SPF record validation also doesn't look like that much of a strong signal for spam filtering either, with 22% of spam mail successfully passing the check.

It's nice that I saw no hard SPF failures for non-spam mail. I checked my inbox for mail that had softfails and permerrors. Some of it was borderline-spammy and some of it was legitimate and appeared to be due to the sender having a misconfigured SPF record.

Another interesting point I noticed is that some sneaky spam mail comes with their own headers claiming SPF evaluation. This might be a problem if the MTA just adds another Received-SPF header at the bottom and doesn't remove the existing one. If you then have a simple filter on Received-SPF: pass somewhere later in the pipeline it's likely the filter will hit the spammer's header first instead of the header your MTA added.

Posted by Tomaž | Categories: Life | Comments »

IEEE 802.11 channel survey statistics

14.02.2016 20:17

I attended the All our shared spectrum are belong to us talk by Paul Fuxjaeger this December at the 32C3. He talked about a kernel patch that adds radio channel utilization data to the beacons sent out by a wireless LAN access point. The patched kernel can be used with OpenWRT to inform other devices in the network of the state of the radio channel, as seen from the standpoint of the access point. One of the uses for this is reducing the hidden node problem and as I understand they used it to improve robustness of a Wi-Fi mesh network.

His patch got my attention because I did not know that typical wireless LAN radios kept such low-level radio channel statistics. I previously did some surveys of the occupancy of the 2.4 GHz band for a seminar at the Institute. My measurements using a TI CC2500 transceiver on a VESNA sensor node showed that the physical radio channel very rarely reached any kind of significant occupancy, even in the middle of very busy Wi-Fi networks. While I did not pursue it further, I remained suspicious of that result. Paul's talk gave me an idea that it would be interesting to compare such measurements with statistics available from the Wi-Fi adapter itself.

I did some digging and found the definition of struct survey_info in include/net/cfg80211.h in Linux kernel:

/**
 * struct survey_info - channel survey response
 *
 * @channel: the channel this survey record reports, mandatory
 * @filled: bitflag of flags from &enum survey_info_flags
 * @noise: channel noise in dBm. This and all following fields are
 *	optional
 * @channel_time: amount of time in ms the radio spent on the channel
 * @channel_time_busy: amount of time the primary channel was sensed busy
 * @channel_time_ext_busy: amount of time the extension channel was sensed busy
 * @channel_time_rx: amount of time the radio spent receiving data
 * @channel_time_tx: amount of time the radio spent transmitting data
 *
 * Used by dump_survey() to report back per-channel survey information.
 *
 * This structure can later be expanded with things like
 * channel duty cycle etc.
 */
struct survey_info {
	struct ieee80211_channel *channel;
	u64 channel_time;
	u64 channel_time_busy;
	u64 channel_time_ext_busy;
	u64 channel_time_rx;
	u64 channel_time_tx;
	u32 filled;
	s8 noise;
};

You can get these values for a wireless adapter on a running Linux system using ethtool or iw without patching the kernel:

$ ethtool -S wlan0
NIC statistics:
(...)
     noise: 161
     ch_time: 19404258681
     ch_time_busy: 3082386526
     ch_time_ext_busy: 18446744073709551615
     ch_time_rx: 2510607684
     ch_time_tx: 371239068
$ iw dev wlan0 survey dump
(...)
Survey data from wlan0
	frequency:			2422 MHz [in use]
	noise:				-95 dBm
	channel active time:		19404338429 ms
	channel busy time:		3082398273 ms
	channel receive time:		2510616517 ms
	channel transmit time:		371240518 ms
(...)

It's worth noting that ethtool will print out values even if the hardware does not set them. For example the ch_time_ext_busy above seems invalid (it's 0xffffffffffffffff in hex). ethtool also incorrectly interprets the noise value (161 is -95 interpreted as an unsigned 8-bit value).

What fields contain valid values depends on the wireless adapter. In fact, grepping for channel_time_busy through the kernel tree lists only a handful of drivers that touch it. For example, the iwlwifi-supported Intel Centrino Advanced-N 6205 in my laptop does not fill any of these fields, while the Atheros AR9550 adapter using the ath9k driver in my wireless router does.

Short of the comment in the code listed above, I haven't yet found any more detailed documentation regarding the meaning of these fields. I checked the IEEE 802.11 standard, but as far as I can see, it only mentions that radios can keep the statistic on the percentage of the time the channel is busy, but doesn't go into details.

Weekly Wi-Fi channel statistic

Weekly Wi-Fi channel noise statistic

I wrote a small Munin plugin that keeps track of these values. For time counters the graph above shows the derivative (effectively the percentage of time the radio spent in a particular state). With some experiments I was able to find out the following:

  • noise: I'm guessing this is the threshold value used by the energy detection carrier sense mechanism in the CSMA/CA protocol. It's around -95 dBm for my router, which sounds like a valid value. It's not static but changes up and down by a decibel or two. It would be interesting to see how the radio dynamically determines the noise level.
  • channel_time: If you don't change channels, this goes up by 1000 ms each 1000 ms of wall-clock time.
  • channel_time_busy: Probably the amount of time the energy detector showed input power over the noise threshold, plus the time the radio was in transmit mode. This seems to be about the same as the sum of RX and TX times, unless there is a lot of external interference (like another busy Wi-Fi network on the same channel).
  • channel_time_ext_busy: This might be a counter for the secondary radio channel used in channel binding, like in 802.11ac. I haven't tested this since channel binding isn't working on my router.
  • channel_time_rx: On my router this increments at around 10% of channel_time rate, even if the network is completely idle, so I'm guessing it triggers at some very low-level, before packet CRC checks and things like that. As expected, it goes towards 100% of channel_time rate if you send a lot of data towards the interface.
  • channel_time_tx: At idle it's around 2% on my router, which seems consistent with beacon broadcasts. It goes towards 100% of channel_time rate if you send a lot of data from the interface.

In general, the following relation seems to hold:

\frac{t_{channel-tx} + t_{channel-rx}}{t_{channel}} < \frac{t_{channel-busy}}{t_{channel}} < 100\%
Posted by Tomaž | Categories: Life | Comments »

Button, button. Who's got the button?

19.01.2016 19:07

As much as I try, I can't switch away from GNOME. Every once in a while I will try to use some other desktop environment in Debian and I invariably switch back after a month. The fact is that for me, GNOME seems to come with fewest little annoyances that need manual fixing or digging through configuration files or source code. Or it might be that I'm just too used to all the small details in how GNOME does things on the user interface side. It's also possible that I understand enough about GNOME internals at this point that when things stop working I usually find the solution pretty quickly.

That does not mean, however, that GNOME no longer manages to surprise me. This is a story about the latest such occurrence. It's a story about a button. This button in particular:

Rotation lock button in GNOME 3.14

I noticed this button after upgrading my work laptop to Debian Jessie that comes with GNOME 3.14. The most curious thing about it was that it does not appear on my home desktop computer with a supposedly identical Debian Jessie setup. What makes my laptop so special that it sprouted an extra button in this prime piece of screen real-estate?

Being the kind of person that first looks into the documentation, I opened the GNOME 3.14 Release Notes. However, there is no note in there about any mysterious new buttons on the click-top-right-corner menu. I was not surprised though, since it might have been added in any of several GNOME versions that were released between now and the previous Debian Stable.

The button is marked with a circular arrow icon. The kind of icon that is often used to distinguish a reboot in contrast to a shutdown when positioned near the power-switch icon. Like for instance on this Ubuntu shutdown dialog:

Screenshot of Ubuntu shutdown dialog.

I don't think it was unreasonable then to assume that this is a reboot button. Anyway, when documentation about a thing fails you, the best next step is usually to poke the thing with a stick. Preferably a long and non-conductive one.

Unfortunately, this approach failed to uncover the purpose of the mysterious button. Upon closer inspection the button did change a bit after clicking on it, but seemingly nothing else happened. On the second thought, it would be kind of weird for GNOME to have a reboot button when at one point it lacked even a shutdown option, for a reason nobody quite understood.

In a final attempt to discover the purpose of the catching-its-own-tail arrow, I hovered the mouse cursor over it. Of course, no helpful yellow strip of text showed up. Tooltips are obsolete now that everything is a smartphone.

At that point I ran out of ideas. Since I had no text associated with the button it seemed impossible to turn to a web search for help. In fact, I realized I don't even know how the menu that holds the button is called these days. I complained about it on IRC and someone suggested doing a Google Image search for the button. This first sounded like a brilliant idea, but it soon turned out that visual search for graphical user interface widgets just isn't there yet:

Results of a search for visually similar images.

Of course, I was stubborn enough to keep searching. In the end, a (text) query led me to a thread on StackExchange that solved the conundrum. I don't remember what I typed into Google when I first saw that result, but once I knew what to search for, the web suddenly turned out to be full of confused GNOME users. Well, at least I was not alone in being baffled by this.

In the end, I only wonder how many of those people that went through this same experience also immediately started swinging their laptop around to see whether the screen on their laptop would actually rotate and were disappointed when, regardless of the screen lock button, the desktop remained stubbornly in the landscape mode.

Posted by Tomaž | Categories: Life | Comments »

End of a project

15.11.2015 11:38

Two weeks ago, the CREW project concluded with a public workshop and a closed meeting with reviewers invited by the European Commission. Thus ended five years of work on testbeds for wireless communications by an international team from Belgium, Ireland, Germany, France and Slovenia. I've been working on the project for nearly four of these years as the technical lead for the Jožef Stefan Institute. While I've been involved in several other projects, the work related to CREW and the testbed clusters we built in Slovenia has occupied most of my time at the Institute.

Spectrum Wars demo at the Wireless Community meeting.

Image by Ingrid Moerman

It's been four years of periodical conference calls, plenary and review meetings, joint experiments at the testbeds, giving talks and demonstrations at scientific conferences, writing deliverables for the Commission, internal reports and articles for publication, chasing deadlines and, of course, the actual work: writing software, developing electronics, debugging both, making measurements in the field and analyzing data, climbing up lamp posts and figuring out municipal power grids from outdated wiring plans.

It's funny how when I'm thinking back, writing is the first thing that comes to mind. I estimate I've written about 20 thousand words of documents per year, which does not seem all that much. It's less than the amount of text I publish yearly on this blog. Counting everyone involved, we produced almost 100 peer-reviewed papers related to the project, which does seem a lot. It has also resulted in my first experience working with a book publisher and a best paper award from the Digital Avionics Systems Conference I have hanging above my desk.

Measuring radio interference inside an Airbus cabin mockup.

Image by Christoph Heller

Remote writing collaboration has definitely been something new for me. It is surprising that what works best in the end are Word documents in emails and lots of manual editing and merging. A web-based document management system helped with keeping inboxes within IMAP quotas, but little else. Some of this is certainly due to technical shortcomings in various tools, but the biggest reason I believe is simply the fact that Word plus email is the lowest common denominator that can be expected of a large group of people of varying professions and organizations with various internal IT policies.

Teleconferencing these days is survivable, but barely so. I suspect Alexander Graham Bell got better audio quality than some of the GoToMeetings I attended. Which is where face-to-face meetings come into the picture. I've been on so many trips during these four years that I've finally came to the point where flying became the necessary evil instead of a rare opportunity to be in an airplane. Most of the meetings have been pleasant, if somewhat exhausting 3 day experiences. It is always nice to meet people you're exchanging emails with in person and see how their institutions look like. Highlights for me were definitely experiments and tours of other facilities. The most impressive that come to mind were seeing Imec RF labs and clean rooms in Leuven, the w-iLab.t Zwijnaarde testbed in Ghent and Airbus headquarters near Munich.

Instrumented Roombas at w-iLab.t

(Click to watch Instrumented Roombas at w-iLab.t video)

One of the closing comments at the review was about identifying the main achievement of the project. The most publicly visible results of the work in Slovenia are definitely little boxes with antennas you can see above streets in Logatec and around our campus. I have somewhat mixed feelings about them. On one hand, it's been a big learning experience. Planning and designing a network, and most of all, seeing how electronics fails when you leave it hanging outside all year round. Designing a programming interface that would be simple enough to learn and powerful enough to be useful. If I would do it again, I would certainly do a lot of things differently, software and hardware wise.

Different kinds of sensor nodes mounted on street lights.

While a number of experiments were done on the testbed, practically all required a lot of hands-on support. We are still far from having a fully automated remote testbed as a service that would be useful to academics and industry alike. Another thing that was very much underestimated was the amount of continuous effort needed to maintain such a testbed in operation. It requires much more than one full-time person to keep something as complex as this up and running. The percentage of working nodes in the network was often not something I could be proud of.

For me personally, the biggest take away from the project has been the opportunity to study practical radio technology in depth - something I didn't get to do much during my undergraduate study. I've had the chance to work with fancy (and expensive) equipment I would not have had otherwise. I studied in detail Texas Instruments CC series of integrated transceivers, which resulted in some interesting hacks. I've designed, built and tested three generations of custom VHF/UHF receivers. These were the most ambitious electronic designs I've made so far. Their capabilities compared to other hardware are encouraging and right now it seems I will continue to work on them in the next year, either as part of my post-graduate studies or under other projects.

SNE-ESHTER analog radio front-end.

I have heard it said several times that CREW was one of the best performing European projects in this field. I can't really give a fair comparison since this is the only such project I've been deeply involved so far. I was disappointed when I heard of servers being shutdown and files deleted as the project wound down, but that is how I hear things usually are when focus shifts to other sources of funding. It is one thing to satisfy own curiosity and desire to learn, but seeing your work being genuinely useful is even better. I learned a lot and got some references, but I was expecting the end result as a whole to be something more tangible, something that would be more directly usable outside of the immediate circle involved in the project.

Posted by Tomaž | Categories: Life | Comments »

TEMPerHUM USB hygrometers

11.10.2015 14:19

Last month at a project meeting in Ghent, Hans Cappelle told me about PCsensor TEMPerHUM hygrometers and thermometers. At around 23 EUR per piece, these are relatively cheap, Chinese made USB-connected sensors. I was looking for something like that to put into a few machines I have running at different places. Knowing the room temperature can be handy. For instance, once I noticed that the CPU and motherboard temperature on one server was rising even though fan RPMs did not change. Was it because the change in the room temperature or was it a sign that heat sinks were getting clogged with dust? Having a reading from a sensor outside of the server's box would help in knowing the reason without having to go and have a look.

TEMPerHUM v1.2 USB hygrometers

When I took them out of the box, I was a bit surprised at their weight. I assumed that the shiny surface was metallized plastic, but in fact it looks like the case is actually made of metal. Each came with a short USB extension cable so you can put it a bit away from the warm computer they are plugged in.

My devices identify as RDing TEMPERHUM1V1.2 on the USB bus (vendor ID 0x0c45, product ID 0x7402). They register two interfaces with the host: a HID keyboard and a HID mouse. The blue circle with TXT in it is actually a button. When you press it, the sensor starts simulating someone manually typing in periodic sensor readouts. As far as I understand, you can set some settings in this mode by playing with Caps lock and Num lock LEDs. I haven't looked into it further though since I don't currently care about this feature, but it is good to know that sensors can be used with only a basic USB HID keyboard driver and a text editor.

Text editor with TEMPerHUM readings typed in.

On Linux, udev creates two hidraw devices for the sensor. To get a sensor readout from those without having to press the button, Frode Austvik wrote a nice little utility. For v1.2 sensors, you need this patched version.

The utility is written in C and is easy to compile for more exotic devices. The only real dependency is HID API. HID API is only packaged for Debian Jessie, but the 0.8.0 source package compiles on Wheezy as well without any modifications. OpenWRT has a package for HID API. The sensor however does not work on my TP-Link router because it uses USB v1.10 and the router's USB ports apparently only support USB v2.x (see OpenWRT ticket #15194).

By default, udev creates hidraw device files that only root can read. To enable any member of the temperhum group to read the sensor, I use the following in /etc/udev/rules.d/20-temperhum.rules:

Update: Previously, I made the hidraw device world-writable. That might be a security problem, although probably a very unlikely one.

SUBSYSTEM=="hidraw", ATTRS{idVendor}=="0c45", ATTRS{idProduct}=="7402", MODE="0660", GROUP="temperhum"

Finally, what about the accuracy? I left one of the sensors (connected via an extension cable) to sit overnight in a room next to an Arduino with a DS18B20 and the old wireless weather monitor I use at home. The two spikes you see on the graphs are from when I opened the windows in the morning.

Temperature versus time for three thermometers.

The DS18B20 has a declared accuracy of ±0.5°C, while the Si7021 sensor that TEMPerHUM is supposedly using should be accurate to ±0.4°C. However, TEMPerHUM is showing between 1.5°C and 2.0°C higher reading than DS18B20. This might be because the USB fob is heating itself a little bit, or it might be some numerical error (or they might be using off-spec chips). Other people have reported that similar sensors tend to overestimate the temperature.

Humidity versus time for two hygrometers.

For relative humidity measurements I don't have a good reference. The difference between my weather monitor and TEMPerHUM is around 10% according to this graph. Si7021 should be accurate to within 3% according to its datasheet. My weather monitor is certainly worse than that. So I can't really say anything about the humidity measurement except that it isn't obviously wrong.

In conclusion, except for the incompatibility with some Atheros SoCs I mentioned above, these sensors seem to be exactly what they say they are. Not super accurate but probably good enough for some basic monitoring of the environment.

Posted by Tomaž | Categories: Life | Comments »

Spectrum Wars

25.07.2015 14:42

SpectrumWars will be a programming game where players compete for bandwidth on a limited piece of radio spectrum. It's also apparently a title that tends to draw attention from bored conference attendees and European commissioners alike.

SpectrumWars XKCD spoof.

with apologies to Randall Munroe

With the CREW project nearing completion, it was decided that one of the last things created in the project will be a game. This might seem an odd conclusion to five years of work from 8 European institutions that previously included little in terms of what most would consider entertainment. We built testing environments for experimentation with modern wireless communications aimed at the exclusive club of people that know what CSMA/CA and PCFICH stand for. In the end, I'm not sure we managed to convince them that what we did was useful to their efforts. Much less I guess the broader public and potential students that would like to study this field.

What better to correct this than to present the need for smart radio testbeds in the form of a fun game? The end of the project is also the time when you want to show off your work in way that reaches the broadest possible audience. Everything to raise the possibility that powers-that-be grant you funding for another project.

In any case, SpectrumWars is a fun little thing to work on and I gladly picked it up in the general lethargy that typically suffuses European projects on their home stretch. The idea stems from a combination of the venerable RobotWar game concept, DARPA Spectrum Challenge and some previous demonstrations by the CREW project at various conferences. Instead of writing a program for a robot that fights off other robots, players program radios. Turning the turret becomes tuning the radio to a desired frequency. A spectrum analyzer replaces the radar and the battle arena is a few MHz of vacant spectrum. Whoever manages to be best at transferring data from their transmitter to their receiver in the presence of competition, wins.

A desk-top setup for SpectrumWars with VESNA sensor nodes.

In contrast to the dueling robots, the radios are not simulated: their transmissions happen in the real world and might share the spectrum with Wi-Fi and the broken microwave next door. To keep it fun, things are simplified, of course. The game abstracts the control of the radios to the bare minimum so that a basic player fits into 10 lines of code, and a pretty sophisticated one into 50. This is not the DARPA Challenge where you need a whole team to have a go (unfortunately, there's no five-figure prize fund either).

So far I've made a hardware-agnostic game controller implementation in Python that takes care of all the dirty little details of running the game and defines the player's interface. The radios are VESNA nodes with Texas Instruments CC2500 transceivers and special firmware, but other CREW partners are currently working on supporting their own hardware. A USRP is used as a spectrum sensor that is shared by all users. While a lot of things are still missing (a user-friendly front-end in particular), it was enough to confirm so far that playing with the algorithms in this way is indeed fun (well, at least for my limited idea of fun).

I gave some more details in a talk I had at the Institute last Friday (slides), or you can go check the draft documentation I wrote or the source, if you feel particularly adventurous.

Title slide from the SpectrumWars talk.

In the end, SpectrumWars is supposed to look like a web site where tournaments between algorithms are continuously performed, where one can submit an entry and where a kind of a scoreboard is held. I'm not optimistic that it will be in fact finished to the point where it will be presentable to the general public in the couple of months left for the project. However I think it would be a shame if it went directly into the bit bucket together with many other things when the project funding runs out.

While the future might be uncertain at the moment, I do have a tentative plan to maybe set up an instance of SpectrumWars at the CCC congress in December. The inherently crowded 2.4 GHz spectrum there would certainly present an interesting challenge for the players and the congress does have a history of unusual programming games.

Posted by Tomaž | Categories: Life | Comments »

Weather radar image artifacts

16.06.2015 21:49

The Slovenian environment agency (ARSO) operates a 5 GHz Doppler weather radar on a hill in the eastern part of the country (Lisca). They publish a map with rate of rainfall estimate based on radar returns each 10 minutes.

Gašper of the opendata.si group has been archiving these rainfall maps for several years now as part of the Slovenian weather API project. A few days ago I've been playing with the part of this archive that I happened to have on my disk. I've noticed some interesting properties of the radar image that show up when you look at statistics generated from many individual images.

This is a sum of approximately 17676 individual rainfall maps, taken between December 2012 and April 2013:

Sum of 17676 weather radar images from ARSO.

Technically what this image shows is how often each pixel indicated non-zero rainfall during this span of time. In other words, the rainiest parts, shown in red, had rain around 18% of the time according to the radar. I ignored the information about the rate of rainfall.

More interesting is how some artifacts become visible:

  • There is a blind spot around the actual location of the radar. The radar doesn't scan above a certain elevation angle and is blind for weather directly above it.

  • There are triangular shadows centered on the radar location. These are probably due to physical obstructions of the radar beam by terrain. I've also heard that some of these radars automatically decrease sensitivity in directions where interference is detected (weather radars share the 5 GHz band with wireless LAN). So it is also possible that those directions have higher noise levels.

  • Concentric bands are clearly visible. The image is probably composed of several 360° sweeps at different elevation angles of the antenna. Although it is curious that their spacing decreases with distance. They might also be result of some processing algorithm that adjusts for the distance in discrete steps.

  • It's obvious that sensitivity is falling with distance. Some parts of the map in the north-western corner never show any rain.

  • Some moiré-like patterns are visible in the darker parts of the image. They are probably the result of interpolation. The map has rectangular pixels aligned with latitude and longitude while the radar produces an image in polar coordinates.

I should mention that ARSO's weather radar system got upgraded in 2014 with a second radar. Newer rainfall maps are calculated from combined data from both radars. They likely have much fewer artifacts. Unfortunately, the format of the images published on the web changed slightly several times during that year, which makes them a bit less convenient to process and compare.

Posted by Tomaž | Categories: Life | Comments »

On cyclostationary detectors in literature

01.03.2015 21:50

Recently, I've been studying cyclostationary detectors. These methods of detecting the presence of a radio transmission seem to be quite popular in academic literature. On the other hand, I kind of glossed over them in my seminar on spectrum sensing. There were several reasons for that: one of them was certainly that I find the theory of cyclostationary signals and statistical spectral analysis hard to wrap my head around. Another reason was however that I couldn't find any papers that would describe a practical cyclostationary detector to sufficient enough detail to allow reproduction of published results. At the very least I wanted to compare its performance against other (non-cyclostationary) detectors in a laboratory experiment before writing about it in any depth.

Here are some of the problems I stumbled upon when trying to understand and reproduce results from scientific papers published on this topic. I don't want to point to any particular work. The example below is just a recent one that crossed my desktop (and happens to nicely illustrate all of the problems I write about). I encountered similar difficulties when reading many other research papers.

Example of unreadable mathematical typesetting.

(from Wireless Microphone Sensing Using Cyclostationary Detector presented at the INCT 2012 conference)

Perhaps the most trivial thing that makes reading unnecessary hard is bad mathematical typesetting. It's sometimes bad enough that it is impossible to follow derivations without first rewriting them to a more consistent form. A part of the blame here goes to the publishers who are preferring manuscripts in Word. I have seen some beautiful articles produced in Word, but the fact is that it is much harder to mess up formatting in LaTeX. Not that it is impossible though. I have had to deal with my share of buggy LaTeX styles.

Another common problem that is simple to fix is inconsistent mathematical notation. For instance the constant Ac2/4 gets dropped somewhere in the middle of the derivation above. Also, x might (or might not) mean two different things. I've heard people argue strongly that such trivial errors are of no concern, since obviously everyone in the field knows what is meant by the author. The problem is that mathematical sloppiness creates confusion for readers that are not intimately familiar with the topic. Why even have a theoretical overview section in the paper then, if the assumption is that it the reader already knows it by heart?

After you manage to bite your way through the theoretical sections to the practical evaluation, it is surprisingly common that various parameters necessary to reproduce the results are left out. For instance, a lot of cyclostationary detectors use a method of estimating the cyclic spectrum called the FFT accumulation method. This method takes several parameters, for instance the sizes of two Fourier transforms that determine its time and frequency resolution. These are often omitted. Another omission is the exact waveform used to test the detector. The paper above gives only frequency deviation used by the wireless microphone, but not the nature of the modulation signal.

Having such difficulty reproducing results from papers based on pure numerical simulations is especially frustrating. I imagine the results are based on a few hundred lines of Matlab code which, if published, would quickly remove any doubts on how the results were obtained. However publishing source together with an article is still a rare exception.


Some of the difficulties I've been having are definitely due to my unfamiliarity with the field. As I keep studying the topic, some of the line of thoughts from the authors who wrote these papers become clearer and some mysteries on how they managed to come up with their results deepen. Undoubtedly some of my own writings aren't as clear, or error free, as I liked them to be. However there are literally hundreds of articles written on the topic of cyclostationary detectors and I'm continuously surprised how much time it is taking me to find a single reproducible example.

Posted by Tomaž | Categories: Life | Comments »

Jahresrückblick

10.01.2015 23:00

I guess too many things happen in a year to be able to mark the whole as good or bad with good conscience. For me, 2014 certainly had had its ups and downs. Looking back at it though, I can't avoid the feeling that the latter were more abundant and that it's been overall a difficult year to weather.

My life in 2014 has been dominated by a bunch of personal problems that surfaced almost exactly at the start of the year. I only wish I could say they are completely behind me. I am not comfortable discussing them here, except to say that I am immensely grateful for the help and support I received, despite the worries and grief I caused. Foremost from my parents and also from professional counselors I met over the past twelve months.

I have completed, even if just barely catching the deadline, the first year of a postgraduate programme at the Jožef Stefan International Postgraduate School. Although I have been encouraged by my employer and others to pursue a PhD, I am not that optimistic that I will complete the study. After being a passive observer of the school for two years and a student for one I'm not convinced that I can make a meaningful contribution to science in this environment. I feel I am too old to sign attendance at lectures, study a topic only for the sake of passing an exam and giving seminar presentations to empty rooms.

Last year has been the time when I completed what is probably my most elaborate electronics project yet. The new UHF receiver for VESNA has been an interesting challenge, a welcome refreshment of some parts of electrical engineering I haven't touched in many years and provided a wonderful sense of achievement when everything fell in place. Still, looking at it now, the success is bitter sweet. It's been work done in an isolated garden. I've done all electronics design, firmware and higher level software development myself and it's too little too late. What I made is a product of European funding and the ecosystem at the Institute. Outside these walls it has precious little advantages over something as mediocre as a rtl-sdr stick in a Raspberry Pi.

Given the above, it's no wonder that 2014 has been kind of dry season for technical side projects. I always tended to have some piled on my table, but this time few deserve mention. I've spent perhaps the most time playing with CubieTruck, getting Debian and a web server running smoothly on it and discovering some of its whims. jsonmerge was another project that had me occupied beyond the hackday that sparked it. While it turned out less generally useful than I initially thought, it was a nice excuse to update my Python skills somewhat.

Speaking of Python, visiting Berlin in summer for EuroPython has definitely been one of the highlights of the year. As was manning the hardware hacking table at the WebCamp Ljubljana.

It's hard to sum up 2014. In some way it's been full of serious work and worries that have at times removed the joy of working in my profession. On the other hand, it's been also about facing problems that went unaddressed for too long and doing things that I might not have imagined doing a few years back. It's hard to say which of them has been a distraction from which. It has left me wishing for change and wanting back times when I could spend evenings hacking away on some fun project without deadlines attached.

Posted by Tomaž | Categories: Life | Comments »

Continuous spectrum monitoring

23.12.2014 19:40

Just before the holidays and with some help from my colleagues at the Institute I managed to finally deploy the first spectrum sensor based on my new UHF receiver design. A handful of similar deployments are planned for the next year. At the end of January I'm traveling to London to mount two similar sensors that will monitor the TV White Spaces pilot in UK as part of a cooperation between King's College London and the CREW project I'm working for.

VESNA stand-alone spectrum sensor in a metal box.

The sensor is based on the VESNA sensor node and is mounted in a small, weather-proof metal box. The previous generation of spectrum sensors (like those that were deployed in our testbed in Logatec) were designed to work in a wireless sensor network. However the low bit rate and reliability of the sensor network and interference caused by another radio next to a sensitive receiver proved to be very limiting. So for this device I went with wired Ethernet even though that reduces the deployment possibilities somewhat. Ethernet now provides a sufficient bit rate to allow for continuous monitoring of the complete UHF band with increased precision that is possible with the new receiver.

In some modes of operation however, the bandwidth from the sensor is still a limiting factor. The Digi Connect ME Ethernet module used here only allows for streaming data up to around 230 kbit/s, which is around 2-3 times slower than what the VESNA's CPU can handle. This is not a problem if you do signal processing on-board the sensor and only send some test statistics back over the network. In that case other things, like the local oscillator and AGC settle time, are usually the limiting factor. However if you want to stream raw signal samples, the network becomes the bottleneck.

Cluster of antennas mounted on JSI building C.

Antenna for the sensor is mounted on top of one of the buildings at the Jožef Stefan Institute campus in Ljubljana. Most of the setup, like power and Ethernet connection, is shared with WLAN Slovenija, who already had some equipment mounted there. On the picture above, my UHF antenna (Super Scan Stick from Moonraker) is second from the left (one with three radials). I'm still using SO-239 connectors and RG-58 coax which I know is not optimal, but they were what I had in stock.

There is direct line of sight to a DVB-T receiver near Ljubljana Castle around 1500 m away and two cellular towers on nearby buildings, so there are plenty of strong transmissions to look at. On the same mast there is also a big omni-directional antenna for 2.4 GHz Wi-Fi. This might cause some interference with sensing in the 800 MHz band (2.4 GHz is the third harmonic of a 800 MHz LO signal), but I don't think that will be a problem. RG-58 cable is bad enough even at sub-1 GHz frequencies and that part of the spectrum seems to be occupied by a strong LTE signal anyway.

Screenshot of the real-time spectrogram plot.

If you click on the image above, you should see a live visualization in you browser of the data stream coming from the sensor.

At the time of writing, the receiver is sweeping the band between 470 MHz and 861 MHz in 1 MHz increments. For each 1 MHz channel, it takes 25000 samples of the baseband signal. From these samples, it calculates the baseband signal power and a vector of sample covariances. It then sends these statistics back to a server that logs them to a hard drive and also provides the visualization (using a somewhat convoluted setup).

The signal power is shown on the spectrogram plot in logarithmic scale. Currently on the spectrogram you can see four DVB-T multiplexes at 482, 522, 562 and 610 MHz and some LTE activity around 800 MHz. Note that the power level is in relative units: 0 dB is the maximum ADC level which can correspond to different absolute power levels due to automatic changes in gain. The sensor can provide calibrated absolute power figures as well, but they are not shown at the moment.

Sample covariances are currently used by a maximum auto-correlation detector to detect whether a channel is vacant or not. This is shown on the live graph below the spectrogram. Note though that this detector works best for narrow-band transmissions, like wireless microphones, and often marks a channel vacant when it is in fact occupied by a wide-band transmission like DVB-T (see my seminar on spectrum sensing methods for details and references).

Posted by Tomaž | Categories: Life | Comments »

Arts & Crafts

19.12.2014 22:44

I have in front of me two sketchbooks. Between them, they contain around a hundred pages and a bundle of loose sheets. Thumbing through them, they start with embarrassingly clumsy pencil sketches, half way through change to line marker drawings and end with digitally shaded print-outs. The first page is dated April 2013. A year and a half later, I'm trying to come up with a reasonable story behind all of this.

The fact is that free-hand drawing is one skill I never really learned. I am pretty handy with a pencil, mind you. I am old enough to have had technical drawing lessons in school that did not involve anything fancier than a compass and a straightedge. I do majority of my notes in a paper notebook and still prefer to do plans and schematics for my home projects with a pen instead of a mouse. However the way one draws objects that do not consist of right angles and straight lines has always eluded me.

Drawing is really hard.

-- Randal Munroe in an recent interview

I have grown up with characters drawn by Miki Muster and I have kept my appetite for comics ever since. I don't think I ever attempted to draw one though. I remember the art class I attended in elementary school. It seems the local artist that held it considered it his job to protect our unspoiled childhood creativity. Any art inspired by popular culture was frowned upon and a reason for a lecture on how completely devoid of imagination our generation was. I guess we stuck to whatever kind of imaginary things children were supposed to draw back then and I was more interested in the room full of ZX Spectrums next door anyway.

So from that point of view, attempting to draw cartoon characters has been an interesting new challenge, something very different from my usual pursuits. Maybe it shouldn't be surprising that I picked up something like that now. I have been writing software at home while studying electronics at the Faculty and I have been doing electronics at home while writing software at work. These days I spend most of my time doing everything from electronics design, software development to writing and giving talks for the Institute. It leaves little space to decompress after work. Doing some non-digital creativity is refreshing.

A working pony, a pony of science.

The other part of this story is, of course, that I've been spending increasing amounts of time on forums and events that have something to do with cartoonish horses and other such things. For better or for worse, it's been a common pass time and means of escape for me this past year. I certainly wouldn't go out and buy a sketch book and a set of pencils if I wouldn't witness a few people from science and engineering professions try themselves in drawing, sometimes with really nice results. It seems that whatever community I happen to meet, I can't manage to stay just a passive observer for long.

If nothing else, that teacher from elementary school was probably right regarding lack of imagination. I don't pretend that what I've made is very original nor that I've managed to teach myself anything else than basics. On the other hand, learning anything necessarily involves some degree of copying. I don't feel this has been a waste even if there is a crowd on the Internet is doing similar things.

The point is that to be truly adept at designing something, you have to understand how it works. [Otherwise] you're drawing ponies. Don't draw ponies.

-- David Kadavy in Design for Hackers

Several years and two schools after that art class I mentioned, I found myself at an art history lecture. It was taught by a bitter old lady who, despite her quite obvious disappointment at the world, was still very much determined to teach us ignorant youth. In addition to explaining historic European architecture styles to us, she also commonly managed to slip in her observations about life in general. On one such occasion she told us how often men fall into the trap of irrationally obsessing over one thing. I remember duly noting it in my notebook.

Maybe this is what she had in mind, maybe not. The fact is that I enjoyed doing it and learning to draw made me look at things around me in new ways. If someone else liked the few results that I shared on the Internet, that much better. Some say it might look terribly silly a few years on, but I'm sure I won't regret gaining a new skill.

Posted by Tomaž | Categories: Life | Comments »