Moving Dovecot indexes and control files

23.12.2016 22:06

Dovecot IMAP server can use a standard Maildir for storage of messages inside user's home directories. The default in that case is to store search indexes and control files in the same directory structure, alongside mail files. That can be convenient, since no special setup is needed and everything is stored in the same place.

However, this doesn't work very well if you have disk quotas enabled on the filesystem that stores Maildirs. In case a user reaches their quota, Dovecot will not be able to write to its own files, which can lead to problems. Hence, documentation recommends that you configure a separate location for Dovecot's files in that case. This is done with INDEX and CONTROL options to the mail_location specification.

For example, after setting up appropriate directory structure and permissions under /var/lib/dovecot:

mail_location = maildir:%h/Maildir:INDEX=/var/lib/dovecot/index/%u:CONTROL=/var/lib/dovecot/control/%u

You can just set this up and leave old index and control files in place. In that case, Dovecot will automatically regenerate them. However, this is not ideal. It can take significant time to regenerate indexes if you have a lot of mail. You also lose some IMAP-related metadata, like message flags and unique IDs, which will confuse IMAP clients. It would be better to move existing files to the new location, however the documentation doesn't say how to do that.

I found that the following script works with Dovecot 2.2.13 on Debian Jessie. As always, be careful when dealing with other people's mail and double check that the script does what you want. I had my share of problems when coming up with this. Make backups.

#!/bin/bash

set -ue

# Run as "migrate_dovecot.sh user path-to-users-maildir" for each user.

# Make sure that Dovecot isn't running or that this specific IMAP user isn't
# connected (and can't connect) while this script runs!
USERNAME=$1
MAILDIR=$2

DOVECOTDIR=/var/lib/dovecot

# Correct after double-checking that this script does what you want.
MV="echo mv -i"

cd $MAILDIR

# Index files like dovecot.index, dovecot.index.cache, etc. go under the 
# INDEX directory. The directory structure should be preserved. For example,
# ~/Maildir/.Foo/dovecot.index should go to index/.Foo/dovecot.index.

# Exception are index files in the root of Maildir. Those should go under 
# .INBOX
b=$DOVECOTDIR/index/$USERNAME/.INBOX
mkdir -p "$b"
$MV *index* "$b"

find . -name "*index*"|while read a; do
	b=$DOVECOTDIR/index/$USERNAME/`dirname "$a"`
	mkdir -p "$b"
	$MV "$a" "$b"
done

# dovecot-uidlist and dovecot-keywords files should go under CONTROL, in a
# similar way to indexes. There is the same exception for .INBOX.
b=$DOVECOTDIR/control/$USERNAME/.INBOX
mkdir -p "$b"
$MV dovecot-uidlist dovecot-keywords "$b"

find . -name "*dovecot*"|while read a; do
	b=$DOVECOTDIR/control/$USERNAME/`dirname "$a"`
	mkdir -p "$b"
	$MV "$a" "$b"
done

# subscriptions file should go to the root of the control directory.

# Note that commands above also move some dovecot-* files into the root of
# the control directory. This seems to be fine.
$MV "subscriptions" "$DOVECOTDIR/control/$USERNAME"
Posted by Tomaž | Categories: Code | Comments »

About the Wire loop probe

15.12.2016 21:08

Recently I was writing about how my father and I were checking a HiFiBerry board for a source of Wi-Fi interference. For want of better equipment we used a crude near-field probe that consisted of a loop of stripped coaxial cable and a trimmer capacitor. We attempted to tune this probe to around 2.4 GHz using the trimmer to get more sensitivity. However we didn't see any effect of capacitance changes on the response in that band.

The probe was made very much by gut feeling, so it wasn't that surprising that it didn't work as expected. We got some interesting results nonetheless. Still, I thought I might do some follow-up calculations to see how far off we were in our estimates of the resonance frequency.

Our probe looked approximately like the following schematic (photograph). The loop diameter was around 25 mm and the wire diameter was around 1 mm. Trimmer capacitor was around 10 pF:

Wire loop at the end of a coaxial cable.

Inductance of a single, circular loop of wire in air is:

L = \mu_0 \frac{D}{2} \left( \ln \frac{8D}{d} - 2 \right) \approx 50 \mathrm{nH}

The wire loop and the capacitor form a series LC circuit. If we ignore the effect of the coaxial cable connection, the resonant frequency of this circuit is:

f = \frac{1}{2 \pi \sqrt{LC}} \approx 200 \mathrm{MHz}

So it appears that we were off by an order of magnitude. In fact, this result is close to the low frequency peak we saw on the spectrum analyzer at around 360 MHz:

Emissions from the HiFiBerry board from DC to 5 GHz.

Working backwards from the equations above, we would need capacitance below 1 pF or loop diameter on the order of millimeters to get resonance at 2.4 GHz. These are very small values. Below 1 pF, stray capacitance of the loop itself would start to become significant and a millimeter-sized loop seems too small to be approximated with lumped elements.

Posted by Tomaž | Categories: Analog | Comments »

HiFiBerry and Wi-Fi interference

01.12.2016 11:43

HiFiBerry is a series of audio output cards designed to sit on the Raspberry Pi 40-pin GPIO connector. I've recently bought the DAC+ pro version for my father to use with a Raspberry Pi 3. He is making a custom box to use as an Internet radio and music player. I picked HiFiBerry because it seemed the simplest, with fewest things that could go wrong (the Cirrus Logic board for instance has many other features in addition to an audio output). It's also well supported out-of-the-box in various Raspberry Pi Linux distributions.

Unfortunately, my father soon found out that the internal wireless LAN adapter on the Raspberry Pi 3 stopped working when HiFiBerry was plugged in. Apparently other people have noticed that as well, as there is an open ticket about it at the Raspberry Pi fork of the Linux kernel.

Several possible causes were discussed on the thread on GitHub, from hardware issues to kernel driver bugs. From those, I found electromagnetic interference the most likely explanation - reports say that the issue isn't always there and depends on the DAC sampling rate and the Wi-Fi channel and signal strength. I thought I might help resolving the issue by offering to make a few measurements with a spectrum analyzer (also, when you have RF equipment on the desk, everything looks like EMI).

HiFiBerry board with a near-field probe over the resonators.

I don't have any near-field probes handy, so we used an ad-hoc probe made from a small wire loop on an end of a coaxial cable. We attempted to tune the loop using a trimmer capacitor to get better sensitivity around 2.4 GHz, but the capacitor didn't have any noticeable effect. We swept this loop around the surface of the HiFiBerry board as well as the Raspberry Pi 3 board underneath.

During these tests, the wireless LAN and Bluetooth interfaces on-board Raspberry Pi were disabled by blacklisting brcmfmac, brcmutil, btbcm and hci_uart kernel modules in /etc/modprobe.d. Apart from this, the Raspberry Pi was booted from an unmodified Volumio SD card image. Unfortunately we don't know what kind of ALSA device settings the Volumio music player used.

What we noticed is that the HiFiBerry board seemed to radiate a lot of RF energy all over the spectrum. The most worrying are spikes approximately 22.6 MHz apart in the 2.4 GHz band that is used by IEEE 802.11 wireless LAN. Note that the peaks on the screenshot below almost perfectly match the center frequencies of channels 1 (2.412 GHz) and 6 (2.437 GHz). The peaks continue to higher frequencies beyond the right edge of the screen and the two next ones match channels 11 and 14. This seems to approximately match the report from Hyperjett about which channels seems to be most affected.

Emissions from the HiFiBerry board in the 2.4 GHz band.

The spikes were highest when the probe was centered around the crystal resonators. This position is shown on the photograph above. This suggests that the oscillators on HiFiBerry are the source of this interference. Phil Elwell mentions some possible I2S bus harmonics, but frequencies we saw don't seem to match those.

Emissions from the HiFiBerry board down to 1 GHz.

Scanning lower frequencies shows that the highest peak is around 360 MHz, but that is likely because of the sensitivity of our probe and not due to something related to the HiFiBerry board.

Emissions from the HiFiBerry board from DC to 5 GHz.

I'm pretty sure these emissions are indeed connected with the HiFiBerry itself. With the probe on Raspberry Pi board underneath HiFiBerry, the spectrum analyzer barely registered any activity. Unfortunately, I forgot to take some measurements with a 2.4 GHz antenna to see how much of this is radiated out into the far-field. I'm guessing not much, since it doesn't seem to affect nearby wireless devices.

Related to that, another experiment points towards the fact that this is an EMI issue. If you connect a Wi-Fi dongle via a USB cable to the Raspberry Pi, it will work reliably as long as the dongle is kept away from the HiFiBerry board. However if you put it a centimeter above the HiFiBerry board, it will lose the connection to the access point.

In conclusion, everything I saw seems to suggest that this is a hardware issue. Unfortunately the design of the HiFiBerry board is not open, so it's hard to be more specific or suggest a possible solution. The obvious workaround is to use an external wireless adapter on an USB extension cable, located as far as feasible from the board.

I should stress though that the measurements we did here are limited by our probe, which was very crude, even compared to a proper home-made one. While frequencies of the peaks are surely correct, the measured amplitudes don't have much meaning. Real EMI testing is done with proper tools in a anechoic chamber, but that is somewhat out of my league at the moment.

Posted by Tomaž | Categories: Analog | Comments »