## GIMP onion layers plug-in

21.02.2017 20:48

Some time ago I was playing with animation making applications on a (non-pro) iPad. I found the whole ecosystem very closed and I had to jump through some hoops to get my drawings back onto a Linux computer. However the fact that you can draw directly on the screen does make some things easier compared to a standalone Wacom tablet, even if the accuracy is significantly worse.

One other thing in particular stood out compared to my old GIMP setup. These applications make it very easy to jump frame by frame through the animation. In one touch you can display the next frame and do some quick edits and then move back with another touch. You can browse up and down the stack as a quick way to preview the animation. They also do something they call onion layering which simply means that they overlay the next and previous frames with reduced opacity so that it's easier to see how things are moving around.

This is all obviously useful. I was doing similar things in GIMP, except that changing frames there took some more effort. GIMP as such doesn't have a concept of frames. Instead you use image layers (or layer groups) as frames. You have to click to select a layer and then a few more clicks to adjust the visibility and opacity for neighboring layers if you want to have the onion layer effect. This quickly amounts to a lot of clicking around if you work on more than a handful of frames.

GIMP does offer a Python plug-in interface however, so automating quick frame jumps is relatively simple. Relatively, because GIMP Python Documentation turns out to be somewhat rudimentary if you're not already familiar with GIMP internals. I found it best to learn from the Python-Fu samples and explore the interface using the built-in interactive console.

The end result of this exercise was the GIMP onion layers plug-in, which you can now find on GitHub together with installation and usage instructions. The plug-in doesn't have much in terms of an user interface - it merely registers a handful of python-fu-onion- actions for stepping to previous or next frame, with or without the onion layer effect. The idea is that you then assign keyboard (or tablet button) shortcuts to these actions. You will have to define the shortcuts yourself though, since the plug-in can't define them for you. I like to use dot and comma keys since they don't conflict with other GIMP shortcuts and match the typical frame step buttons on video players.

If you follow the layer structure suggested by the Export layers plug-in, this all works quite nicely, including handling of background layers. The only real problem I encountered was the fact that the layer visibility and opacity operations clutter the undo history. Unfortunately, that seems to be the limitation of the plug-in API. Other plug-ins work around this by doing operations on a duplicate of the image, but obviously I can't do that here.

I should note that I was using GIMP 2.8.14 from Debian Jessie, so the code might be somewhat outdated compared to latest GIMP 2.8.20. Feedback in that regard is welcome, as always.

Posted by | Categories: Code | Comments »

## Python applications in a .zip

08.02.2017 10:20

Poking around youtube-dl I found this interesting recipe on how to package a self-contained Python application. Youtube-dl ships as a single executable file you can run immediately, or put somewhere into your PATH. This makes it very convenient to use even when you don't want to do the usual pip install dance. Of course, it comes at the cost of not resolving any dependencies for you.

I was expecting the file to be a huge, monolithic Python source file, but in fact it's a ZIP with a prepended hash-bang and nicely structured Python package inside. Simplified a bit, here is the gist of the Makefile part that builds it:

hello:
cd src && zip ../hello hello/*.py __main__.py
echo '#!/usr/bin/python' > hello
cat hello.zip >> hello
rm hello.zip
chmod a+x hello


Now, if src/__main__.py contains:

import hello
hello.greet()


And src/hello/__init__.py contains:

def greet():
print("Hello, World!")


Building the executable and running hello from the command-line should result in the standard greeting:

$make cd src && zip ../hello hello/*.py __main__.py adding: hello/__init__.py (stored 0%) adding: __main__.py (stored 0%) echo '#!/usr/bin/python' > hello cat hello.zip >> hello rm hello.zip chmod a+x hello$ ./hello
Hello, World!


How does this work? Apparently it's quite an old trick with some added refinement. Already since version 2.3 Python knows how to import modules directly from ZIP files in the same way as from the usual directories. Python also allows executing modules from the command-line.

It sounds very much like Java JARs, doesn't it? The only missing part is the #!... line that makes the Linux kernel use the Python interpreter when executing the file. Since ZIP format ignores any junk that precedes the compressed data, the line can simply be prepended as if the whole file was a simple Bash script.

Posted by | Categories: Code | Comments »

## Jahresrückblick

29.01.2017 18:56

So this has been 2016. Another year that went by all too fast. It has made the world a bit more broken and a whole lot more selfish and incomprehensible.

I haven't traveled much. Except for one project meeting I've hardly left the country. In part because other colleagues took the burden of attending meetings where the Institute had to have a presence. The other part must have been the effect of attacks around Europe and the newly erected borders and barbed wires. I haven't been to any foreign conference, I've skipped 33C3 and I had to cancel my visit to GalaCon at the last moment.

On the other hand, it feels like the local tech meetups have mostly devolved into TED-style talks. I miss the old grassroots events where people would discuss technologies they found interesting or fascinating projects that grew in their basements. It seems the only valid reason to be excited about something these days is if you want to be hired or want to hire someone.

Last year more than before I found myself in situations where people were willing to talk collaboration up to the point where they got what they wanted - and then walked away without as much as a thank you. The charge more mantra is now being taught to every STEM student as a solution to this, and I find it increasingly depressing. Must you really state an hourly rate before even saying hello?

Social events now have codes of conduct, but seem more hostile than ever. Then again, maybe that tells more about myself than those events. Sometimes I feel like I don't understand people and these communities anymore.

All this obviously affected my motivation to work on projects in my free time. Existing Free software I maintain has mostly been on life support last year. Some things I would usually took the effort to clean and publish as open source were left on my disk. I've done very little electronics work outside of my duties at the Institute. In general I stepped away a lot from the usual technical stuff. I rather spent evenings reading, and at a drawing class in the National Gallery.

I'm now in my 6th year employed as a research assistant at the Institute. My project work last year has been mostly related to ultra-narrowband technology. I have 26 publications listed in the national bibliographic database and political games in academia still manage to catch me completely by surprise. With the start of the new academic year I've made a decision to finish my PhD and set a deadline to wrap things up by October.

Most of my time after that have been dedicated to concluding my research work, and writing journal papers. My doctoral topic application has been accepted, but I'm still short of the publication quota. A paper I've been trying to publish since April 2014 has since then accumulated more than 200 commits to the LaTeX source and is currently under yet another peer review.

I don't want to make any big plans for 2017. I have plenty of ideas for interesting projects and a backlog of (non-academic) things I would like to write about. However, given how much work I still have ahead of me before my viva voce, I doubt much of that will happen this year.

Posted by | Categories: Life | Comments »

## Moving Dovecot indexes and control files

23.12.2016 22:06

Dovecot IMAP server can use a standard Maildir for storage of messages inside user's home directories. The default in that case is to store search indexes and control files in the same directory structure, alongside mail files. That can be convenient, since no special setup is needed and everything is stored in the same place.

However, this doesn't work very well if you have disk quotas enabled on the filesystem that stores Maildirs. In case a user reaches their quota, Dovecot will not be able to write to its own files, which can lead to problems. Hence, documentation recommends that you configure a separate location for Dovecot's files in that case. This is done with INDEX and CONTROL options to the mail_location specification.

For example, after setting up appropriate directory structure and permissions under /var/lib/dovecot:

mail_location = maildir:%h/Maildir:INDEX=/var/lib/dovecot/index/%u:CONTROL=/var/lib/dovecot/control/%u


You can just set this up and leave old index and control files in place. In that case, Dovecot will automatically regenerate them. However, this is not ideal. It can take significant time to regenerate indexes if you have a lot of mail. You also lose some IMAP-related metadata, like message flags and unique IDs, which will confuse IMAP clients. It would be better to move existing files to the new location, however the documentation doesn't say how to do that.

I found that the following script works with Dovecot 2.2.13 on Debian Jessie. As always, be careful when dealing with other people's mail and double check that the script does what you want. I had my share of problems when coming up with this. Make backups.

#!/bin/bash

set -ue

# Run as "migrate_dovecot.sh user path-to-users-maildir" for each user.

# Make sure that Dovecot isn't running or that this specific IMAP user isn't
# connected (and can't connect) while this script runs!
USERNAME=$1 MAILDIR=$2

DOVECOTDIR=/var/lib/dovecot

# Correct after double-checking that this script does what you want.
MV="echo mv -i"

cd $MAILDIR # Index files like dovecot.index, dovecot.index.cache, etc. go under the # INDEX directory. The directory structure should be preserved. For example, # ~/Maildir/.Foo/dovecot.index should go to index/.Foo/dovecot.index. # Exception are index files in the root of Maildir. Those should go under # .INBOX b=$DOVECOTDIR/index/$USERNAME/.INBOX mkdir -p "$b"
$MV *index* "$b"

find . -name "*index*"|while read a; do
b=$DOVECOTDIR/index/$USERNAME/dirname "$a" mkdir -p "$b"
$MV "$a" "$b" done # dovecot-uidlist and dovecot-keywords files should go under CONTROL, in a # similar way to indexes. There is the same exception for .INBOX. b=$DOVECOTDIR/control/$USERNAME/.INBOX mkdir -p "$b"
$MV dovecot-uidlist dovecot-keywords "$b"

find . -name "*dovecot*"|while read a; do
b=$DOVECOTDIR/control/$USERNAME/dirname "$a" mkdir -p "$b"
$MV "$a" "$b" done # subscriptions file should go to the root of the control directory. # Note that commands above also move some dovecot-* files into the root of # the control directory. This seems to be fine.$MV "subscriptions" "$DOVECOTDIR/control/$USERNAME"

Posted by | Categories: Code | Comments »

## About the Wire loop probe

15.12.2016 21:08

Recently I was writing about how my father and I were checking a HiFiBerry board for a source of Wi-Fi interference. For want of better equipment we used a crude near-field probe that consisted of a loop of stripped coaxial cable and a trimmer capacitor. We attempted to tune this probe to around 2.4 GHz using the trimmer to get more sensitivity. However we didn't see any effect of capacitance changes on the response in that band.

The probe was made very much by gut feeling, so it wasn't that surprising that it didn't work as expected. We got some interesting results nonetheless. Still, I thought I might do some follow-up calculations to see how far off we were in our estimates of the resonance frequency.

Our probe looked approximately like the following schematic (photograph). The loop diameter was around 25 mm and the wire diameter was around 1 mm. Trimmer capacitor was around 10 pF:

Inductance of a single, circular loop of wire in air is:

L = \mu_0 \frac{D}{2} \left( \ln \frac{8D}{d} - 2 \right) \approx 50 \mathrm{nH}

The wire loop and the capacitor form a series LC circuit. If we ignore the effect of the coaxial cable connection, the resonant frequency of this circuit is:

f = \frac{1}{2 \pi \sqrt{LC}} \approx 200 \mathrm{MHz}

So it appears that we were off by an order of magnitude. In fact, this result is close to the low frequency peak we saw on the spectrum analyzer at around 360 MHz:

Working backwards from the equations above, we would need capacitance below 1 pF or loop diameter on the order of millimeters to get resonance at 2.4 GHz. These are very small values. Below 1 pF, stray capacitance of the loop itself would start to become significant and a millimeter-sized loop seems too small to be approximated with lumped elements.

Posted by | Categories: Analog | Comments »

## HiFiBerry and Wi-Fi interference

01.12.2016 11:43

HiFiBerry is a series of audio output cards designed to sit on the Raspberry Pi 40-pin GPIO connector. I've recently bought the DAC+ pro version for my father to use with a Raspberry Pi 3. He is making a custom box to use as an Internet radio and music player. I picked HiFiBerry because it seemed the simplest, with fewest things that could go wrong (the Cirrus Logic board for instance has many other features in addition to an audio output). It's also well supported out-of-the-box in various Raspberry Pi Linux distributions.

Unfortunately, my father soon found out that the internal wireless LAN adapter on the Raspberry Pi 3 stopped working when HiFiBerry was plugged in. Apparently other people have noticed that as well, as there is an open ticket about it at the Raspberry Pi fork of the Linux kernel.

Several possible causes were discussed on the thread on GitHub, from hardware issues to kernel driver bugs. From those, I found electromagnetic interference the most likely explanation - reports say that the issue isn't always there and depends on the DAC sampling rate and the Wi-Fi channel and signal strength. I thought I might help resolving the issue by offering to make a few measurements with a spectrum analyzer (also, when you have RF equipment on the desk, everything looks like EMI).

I don't have any near-field probes handy, so we used an ad-hoc probe made from a small wire loop on an end of a coaxial cable. We attempted to tune the loop using a trimmer capacitor to get better sensitivity around 2.4 GHz, but the capacitor didn't have any noticeable effect. We swept this loop around the surface of the HiFiBerry board as well as the Raspberry Pi 3 board underneath.

During these tests, the wireless LAN and Bluetooth interfaces on-board Raspberry Pi were disabled by blacklisting brcmfmac, brcmutil, btbcm and hci_uart kernel modules in /etc/modprobe.d. Apart from this, the Raspberry Pi was booted from an unmodified Volumio SD card image. Unfortunately we don't know what kind of ALSA device settings the Volumio music player used.

What we noticed is that the HiFiBerry board seemed to radiate a lot of RF energy all over the spectrum. The most worrying are spikes approximately 22.6 MHz apart in the 2.4 GHz band that is used by IEEE 802.11 wireless LAN. Note that the peaks on the screenshot below almost perfectly match the center frequencies of channels 1 (2.412 GHz) and 6 (2.437 GHz). The peaks continue to higher frequencies beyond the right edge of the screen and the two next ones match channels 11 and 14. This seems to approximately match the report from Hyperjett about which channels seems to be most affected.

The spikes were highest when the probe was centered around the crystal resonators. This position is shown on the photograph above. This suggests that the oscillators on HiFiBerry are the source of this interference. Phil Elwell mentions some possible I2S bus harmonics, but frequencies we saw don't seem to match those.

Scanning lower frequencies shows that the highest peak is around 360 MHz, but that is likely because of the sensitivity of our probe and not due to something related to the HiFiBerry board.

I'm pretty sure these emissions are indeed connected with the HiFiBerry itself. With the probe on Raspberry Pi board underneath HiFiBerry, the spectrum analyzer barely registered any activity. Unfortunately, I forgot to take some measurements with a 2.4 GHz antenna to see how much of this is radiated out into the far-field. I'm guessing not much, since it doesn't seem to affect nearby wireless devices.

Related to that, another experiment points towards the fact that this is an EMI issue. If you connect a Wi-Fi dongle via a USB cable to the Raspberry Pi, it will work reliably as long as the dongle is kept away from the HiFiBerry board. However if you put it a centimeter above the HiFiBerry board, it will lose the connection to the access point.

In conclusion, everything I saw seems to suggest that this is a hardware issue. Unfortunately the design of the HiFiBerry board is not open, so it's hard to be more specific or suggest a possible solution. The obvious workaround is to use an external wireless adapter on an USB extension cable, located as far as feasible from the board.

I should stress though that the measurements we did here are limited by our probe, which was very crude, even compared to a proper home-made one. While frequencies of the peaks are surely correct, the measured amplitudes don't have much meaning. Real EMI testing is done with proper tools in a anechoic chamber, but that is somewhat out of my league at the moment.

Posted by | Categories: Analog | Comments »

## AFG-2005 noise generation bug

09.10.2016 13:49

The noise output function in the GW Instek AFG-2005 arbitrary function generator appears to have a bug. If you set up the generator to output a sine wave at 10 kHz, and then switch to noise output using the FUNC key, you get output like this:

The red trace shows the spectrum of the signal from the signal generator using the FFT function of the oscilloscope. The yellow trace shows the signal in the time domain.

The noise spectrum looks flat and starts to roll off somewhere beyond 10 MHz. This is what you would expect for an instrument that is supposed to have a 20 MHz DAC. However, if you set the output to a sine wave at 1 kHz before switching to noise output, the signal looks significantly different:

These two later oscilloscope screenshots have been made using the same settings as the pair above.

This is obviously an error on the part of the signal generator. The setting for the sine wave output shouldn't affect the noise output. It looks like the DAC is now only set to around 4 MHz sampling rate. Probably it has been switched to a lower sampling rate for the low-frequency sine wave output and the code forgot to switch it back for the noise function.

This behavior is perfectly reproducible. If you switch back to sine wave output, increase the frequency to 10 kHz or more and switch to noise output, the DAC sampling rate is increased again. Similarly, if you set a 100 Hz sine wave, the DAC sampling rate is set to 400 kHz. As far as I can see there is no mention of this in the manual and you cannot set the sampling rate manually. The FREQ button is disabled in Noise mode and there is no indication on the front panel about which sampling rate is currently used.

I've been writing about the AFG-2005 before. It's an useful instrument, but things like this make it absolutely necessary to always have an oscilloscope at hand to verify that the generator is actually doing what you think it should be doing.

Posted by | Categories: Life | Comments »

## A naive approach to rain prediction

02.10.2016 19:49

Ever since the Slovenian environment agency started publishing Doppler weather radar imagery I've been a regular visitor on their web site. For the last few years I try to use my bicycle for my daily commute as much as possible. However, I'm not such a hard-core fan of biking that I would do it in any weather. The animated map with the recent rainfall estimate history helps very much with that: it's quite easy to judge by eye whether there will be rain in the next 30 minutes or so and hence whether to seek alternative modes of transportation.

Some time ago I had a more serious (or should I say scientific) use for weather data, related to one of the projects at the Institute. Gašper helpfully provided me with some historical archives then and I also started collecting images myself straight from ARSO website. That project came and went, but the amount of data on my disk kept growing. I've been continuously tempted to do something interesting with it. I've previously written what the data seems to reveal about the radar itself.

Most of all, I've been playing with the idea of doing that should-I-take-the-bus prediction automatically. Obviously I'm not a weather expert, so my experiments were very naive from that perspective. For instance, a while ago I tried estimating an optical flow field from the apparent movement of regions with rain and then using that to predict their movement in the near future. That didn't really work. Another idea I had was to simply dump the data into a naive Bayesian model. While that also didn't work to any useful degree as far as prediction is concerned, it did produce some interesting results worth sharing.

What I did was model rain at any point in the radar image (x, y) and time t as a random event:

R_{x,y,t}

To determine whether the event happened or not from the historical data, I simply checked whether the corresponding pixel was clear or not - I ignored the various rates of rainfall reported. To calculate the prior probability of rain on any point on the map, I did a maximum-likelihood estimation:

P(R_{x,y}) = \frac{n_{x,y}}{N}

Here, nx,y is number of images where the point shows rain and N is the total number of images in the dataset.

I was interested in predicting rain at one specific target point (x0, y0) based on recent history of images. Hence I estimated the following conditional probability:

P(R_{x_0,y_0,t+\Delta t}|R_{x,y,t})

This can be estimated from historical data in a similar way as the prior probability. In other words, I was interested in how strongly is rain at some arbitrary point on the map x,y at time t related to the fact that it will rain at the target point at some future time tt. Are there any points on the map that, for instance, always show rain 30 minutes before it rains in Ljubljana?

The video below shows this conditional probability for a few chosen target points around Slovenia (marked by small white X). Brighter colors show higher conditional probability and hence stronger relation. The prior probability is shown in lower-right corner. In the video, the time difference runs in reverse, from 4 hours to 0 in 10 minute steps. For each point, the animation is repeated 4 times before moving to the next point.

The estimates shown are calculated from 62573 radar images recorded between July 2015 and September 2016. Since the format of the images has been changing over time it's hard to integrate older data.

As you might expect, when looking several hours into the future, there is very little relation between points on the map. All points are dull red. If it rains now in the east, it might rain later in Ljubljana. Similarly, if it rains in the west. There's no real difference - the only information you get is that it's generally rainy weather in the whole region.

When you decrease the time difference, you can see that nearby points are starting to matter more than those far away. Brighter colors concentrate around the target. Obviously, if it rains somewhere around Ljubljana, there's a higher chance it will shortly rain in Ljubljana as well. If you note the color scale though, it's not a particularly strong relation unless you're well below one hour.

What's interesting is that for some cities you can see that rain more often comes from a certain direction. Around the coast and in Notranjska region, the rain clouds seem to mostly move in from the Adriatic sea (lower-left corner of the map). This seems to fit the local saying you can sometimes hear, that the "weather comes in from the west". In the mountains (top-left), it seems to come from the north. All this is just based on one year of historical data though, so it might not be generally true over longer periods.

Of course, such simple Bayesian models are horribly out-of-fashion these days. A deep learning convolutional neural network might work better (or not), but alas, I'm more or less just playing with data on a rainy weekend and trying to remember machine learning topics I used to know. There's also the fact that ARSO themselves now provide a short-term rain prediction through an application on their website. It's not the easiest thing to find (Parameter Selection - Precipitation in the menu and then Forecast on the slider below). I'm sure their models are way better than anything an amateur like me can come up with, so I doubt I'll spend any more time on this. I might try to add the forecast to ARSO API at one point though.

Posted by | Categories: Life | Comments »

## Blacklisting users for inbound mail in Exim

18.09.2016 12:00

You can prevent existing local users from receiving mail by redirecting them to :fail: in /etc/aliases. For example, to make SMTP delivery to list@... fail with 550 Unrouteable address:

list: :fail:Unrouteable address


See special items section in the Redirect router documentation.

By default, Exim in Debian will attempt to deliver mail for all user accounts, even non-human system users. System users (like list above) typically don't have a traditional home directory set in /etc/passwd. This means that mail for them will get stuck in queue as Exim tries and fails to write to their mailbox. Because spam also gets sent to such addresses, mail queue will grow and various things will start to complain. Traditionally, mail for system accounts is redirected to root in /etc/aliases, but some accounts just receive a ton of spam and it's better to simply reject mail sent to them.

Another thing worth pointing out is the Handling incoming mail for local accounts with low UID section in README.Debian.gz in case you want to reject mail sent to all system accounts.

This took way too much time to figure out. There's a ton of guides on how to blacklist individual users for outgoing mail, but none I could find for cases like this. I was half-way into writing a custom router before I stumbled upon this feature.

Posted by | Categories: Code | Comments »

## Script for setting up multiple terminals.

16.09.2016 19:04

Sometimes I'm working on software that requires running a lot of different inter-dependent processes (are microservices still a thing?). Using systemd or some other init system for starting up such systems is fine for production. While debugging something on my laptop however it's useful to have each process running in its own X terminal. This allows me to inspect any debug output and to occasionally restart something. I used to have scripts that would run commands in individual GNU screen sessions, but that had a number of annoying problems.

I've recently came up with the following:

#!/bin/bash

set -ue

if [ "$#" -ne 1 ]; then echo "USAGE:$0 path_to_file"
echo "File contains one command per line to be started in a terminal window."
exit 1
fi

cat "$1" | while read CMD; do if [ -z "$CMD" -o "${CMD:0:1}" = "#" ]; then continue fi RCFILE=tempfile cat > "$RCFILE" <<END
source ~/.bashrc
history -s $CMD echo$CMD
$CMD END gnome-terminal -e "/bin/bash --rcfile \"$RCFILE\""
rm "\$RCFILE"
done


This script reads a file that contains one command per line. Empty lines and lines starting with a hash sign are ignored. For each line it opens a new gnome-terminal (adjust as needed - most terminal emulators support the -e argument) and runs the command in a way that:

• The terminal doesn't immediately close after the command exits. Instead it drops back to bash. This allows you to inspect any output that got printed right before the process died.
• The command is printed on top of the terminal before the command runs. This allows you to identify the terminal running a particular process in case that is not obvious from the command output. For some reason, gnome-terminal's --title doesn't work.
• The command is appended on top of bash' history list. This allows you to easily restart the process that died (or you killed with Ctrl-C) by simply pressing the up cursor and enter keys.
Posted by | Categories: Code | Comments »