When I was processing the raw measurement results from Munich experiment I noticed that absolute timestamps on spectrograms recorded by two VESNA nodes were differing randomly from zero to three seconds and required manual alignment.
The experimental setup was not expected to give very precise time reference: each VESNA has its own free running real-time clock stabilized by a 32.768 kHz quartz which was providing time relative to the start of the measurement. Absolute time on the other hand was provided by two Linux running laptops to which the sensor nodes were sending the data. So all in all the accuracy of our timestamps depended on four quartz clocks.
Nevertheless this result was surprising to me. I expected these four clocks to drift apart in the three hours of measurements, but this drift should be uniform. I can't explain what could cause the difference between clocks to randomly change between experiments.
To rule out any problems on VESNA I rigged up a simple test where I compared the difference between VESNA's real-time clock and the clock provided by the Linux kernel with a running NTP daemon. This is the result of four test runs with two nodes that we were using in Munich:
As you can see, the clocks do drift apart and do so more or less linearly (nodes were turned off for the night before each test to also see if warm-up affected the drift). Node 117 drifts at around 3 ppm and node 113 at 9 ppm. This is actually quite bad, as node 117 would gain a second each four days, even considering that the reference here was a laptop that probably isn't a shining example of accuracy either.
The only weird thing is the strange bump on the graph for node 113, where drift was positive for around 15 minutes and then turned back to negative. I could not reproduce it in any of the later runs and it might be due to NTP adjusting the laptops clock. But even that can't explain the observed deviations that are more than ten times what we see here.
While this is of course not conclusive, it does give me some confidence that it might be the PC software that was causing these problems.