Follow up on Atmel ZigBit modules

27.08.2014 12:09

I've ranted before about the problematic Atmel ZigBit modules and the buggy SerialNet firmware. In my back-of-the envelope analysis of failure modes in the Jožef Stefan Institute's sensor networks, one particular problem stood out related to this low-powered mesh networking hardware: a puzzling failure that prevents a module from joining the mesh and can seemingly be fixed by reprogramming the module's firmware.

A week ago Adam posted a link to the SerialNet source in a comment to my old blog post. While I've mostly moved on to other things, this new piece of information gave me sufficient excuse to spend another few hours exploring this problem.

Atmel ATZB_900_B0 module on a VESNA SNR-MOD board.

A quick look around Atmel's source package revealed that it contains only the code for the serial interface to the underlying proprietary ZigBee stack. There are no low-level hardware drivers for the radio and no actual network stack code in there. It didn't seem likely that the bug I was hunting was caused by this thin AT-command interface code. On the other hand, this code could be responsible for dropping out characters in the serial stream. However we have sufficient workarounds in place for that bug and it's not worth spending more time on it.

One thing caught my eye in the source: ATPEEK and ATPOKE commands. ATZB-900-B0 modules consist of an ATmega1281 microcontroller and an AT86TF212 transceiver. These two commands allow for raw access to the radio hardware registers, microcontroller RAM, code flash ROM and configuration EEPROM. I thought that given these, maybe I could find out what gets corrupted in module's non-volatile memories and perhaps fix it through the AT-command interface.

Only after figuring out how to use them from studying the source, I found out that these two commands have been in fact documented in revision 8369B of the SerialNet User Guide. Somehow I overlooked this addition previously.


For the sake of completeness, here is a more detailed description of the problem:

A module that previously worked fine and passed all of my system tests will suddenly no longer respond to an AT+WJOIN command. It will not respond with either OK nor ERROR (or their numeric equivalents). However it will respond to other commands in a normal fashion. This can happen after the module has been deployed for several months or only after a few hours.

A power cycle, reset or restoring factory defaults does not fix this. The only known way of restoring the module is to reprogram its firmware through the serial port using Atmel's Bootloader PC Tool for Windows. This reprogramming invokes a bootloader mode on the module and refreshes the contents of the microcontroller's flash ROM as well as resets the configuration EEPROM contents.

It appears that this manifests more often with sensor nodes that are power-cycled regularly. However, in our setup a node only joins the network once after a power-cycle. Even if the bug is caused by some random event that happens anytime during the uptime of the node, it will not be noticed until the next power cycle. So it is possible that it's not the power cycling itself that causes the problem. Aggressive power-cycling tests as well don't seem to increase the occurrence of the bug.


So, with the new found knowledge of ATPEEK I dumped the contents of the EEPROM and flash ROM from two known-bad modules and a few good ones. Comparing the dumps revealed that both of the bad modules are missing a 256 byte block of code from the flash starting at address 0x00011100:

--- zb_046041_good_flash.hex	2014-08-25 16:41:51.000000000 +0200
+++ zb_046041_bad_flash.hex	2014-08-25 16:41:47.000000000 +0200
@@ -4362,22 +4362,8 @@
 000110d0  88 23 41 f4 0e 94 40 88  86 e0 80 93 d5 13 0e 94  |.#A...@.........|
 000110e0  f0 7b 1c c0 80 91 da 13  88 23 99 f0 82 e2 61 ee  |.{.......#....a.|
 000110f0  73 e1 0e 94 41 0c 81 e0  80 93 e5 13 8e e3 91 e7  |s...A...........|
-00011100  90 93 e7 13 80 93 e6 13  8b ed 93 e1 0e 94 86 14  |................|
-00011110  05 c0 0e 94 9a 70 88 81  0e 94 b5 70 df 91 cf 91  |.....p.....p....|
-00011120  08 95 fc 01 80 81 88 23  29 f4 0e 94 40 88 0e 94  |.......#)...@...|
-00011130  e1 71 08 95 0e 94 b5 70  08 95 a2 e1 b0 e0 e3 ea  |.q.....p........|
-00011140  f8 e8 0c 94 71 f4 80 e0  94 e0 90 93 c7 17 80 93  |....q...........|
-00011150  c6 17 0e 94 0f 78 80 91  d9 13 83 70 83 30 61 f1  |.....x.....p.0a.|
-00011160  88 e2 be 01 6f 5f 7f 4f  0e 94 41 0c 89 81 88 23  |....o_.O..A....#|
-00011170  19 f1 0e 94 a6 9f 6b e1  70 e1 48 e0 50 e0 0e 94  |......k.p.H.P...|
-00011180  47 f5 8c 01 8b e2 be 01  6e 5f 7f 4f 0e 94 41 0c  |G.......n_.O..A.|
-00011190  8a 81 88 23 19 f0 01 15  11 05 71 f4 8e 01 0d 5f  |...#......q...._|
-000111a0  1f 4f 87 e2 b8 01 0e 94  41 0c c8 01 60 e0 0e 94  |.O......A...`...|
-000111b0  22 5c 80 e0 0e 94 8d 5c  80 91 d9 13 81 ff 14 c0  |"\.....\........|
-000111c0  0e 94 40 88 0e 94 68 67  80 91 40 10 87 70 19 f4  |..@...hg..@..p..|
-000111d0  0e 94 e1 71 15 c0 81 50  82 30 90 f4 86 e0 80 93  |...q...P.0......|
-000111e0  d5 13 0e 94 f0 7b 0c c0  80 91 40 10 87 70 19 f4  |.....{....@..p..|
-000111f0  0e 94 bf 71 05 c0 81 50  82 30 10 f4 0e 94 df 70  |...q...P.0.....p|
+00011100  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
+*
 00011200  62 96 e4 e0 0c 94 8d f4  80 91 d7 13 90 91 d8 13  |b...............|
 00011210  00 97 d9 f4 84 e5 97 e7  90 93 d8 13 80 93 d7 13  |................|
 00011220  80 91 40 10 87 70 82 30  11 f4 0e 94 04 77 0e 94  |..@..p.0.....w..|

This is puzzling for several reasons.

First of all, it seems unlikely that this is a hardware problem. Both bad modules (with serial numbers apart by several 10000s) had lost the same block of code. If flash lost its contents due to out-of-spec voltage during programming or some other hardware problem I would expect the bad address to be random.

However a software bug causing a failure like that seems highly unlikely as well. I was expecting to see some kind of an EEPROM corruption. EEPROM is used to persistently store module settings and I assume the firmware often writes to it. However flash ROM should be mostly read-only. I find it hard to imagine what kind of a bug could erase a block - reprogramming the flash on a microcontroller is typically a somewhat complicated procedure that is unlikely to come up by chance.

One possibility is that we are maybe somehow unknowingly invoking the bootloader mode during the operation of the sensor node. During my testing however I found out that just invoking the serial bootloader mode without also supplying it with a fresh firmware image corrupts the flash ROM sufficiently that the module does not boot at all. The Bootloader PC Tool seems to suggest that these modules also have some kind of an over-the-air upgrade functionality, but I haven't yet looked into how that works. It's possible we're enabling that somehow.

Unfortunately, the poke functionality does not allow you to actually write to flash (you can write to RAM and EEPROM though). So even if this currently allows me to detect a corrupt module flash while the node is running, that is only good for saying that the module won't come back on-line after a reboot. I can't fix the problem without fully reprogramming the firmware. This means either hooking the module to a laptop or implementing the reprogramming procedure on the sensor node itself. The latter is not trivial, because it involves implementing the programming protocol and somehow arranging for the storage of a complete uncorrupted SerialNet firmware image on the sensor node.

Posted by Tomaž | Categories: Code | Comments »

jsonmerge

20.08.2014 21:01

As I mentioned in my earlier post, my participation at the Open Contracting code sprint during EuroPython resulted in the jsonmerge library. After the conference I slowly cleaned up the remaining few issues and brought up code coverage of unit tests to 99%. The first release is now available from PyPi under the MIT license.

jsonmerge tries to solve a problem that seems simple at first: given a series of structured JSON documents, how to create a single document that contains an aggregate of all their contents. With simple documents that might be as trivial as calling an update() method on a dict:

>>> a = {'foo': 1}
>>> b = {'bar': 2}

>>> c = a.update(b)
>>> c
{'foo': 1, 'bar': 2}

However, even with just two plain dictionaries, things can quickly get complicated. What should happen if both documents contain a field with the same name? Should a later value overwrite the earlier one? Or should the resulting document have in that place a list that contains both values? Source JSON documents themselves can also contain arrays (or arrays of arrays) and handling those is even less straightforward than dictionaries in this example.

Often I've seen a problem like this solved in application code - it's relatively simple to encode your wishes in several hundreds lines of Python. However JSON is a very flexible format while such code is typically brittle. Change the input document a bit and more often than not your code will start throwing KeyErrors left and right. Another problem with this approach is that it's often not obvious from the code what kind of a strategy is taken for merging changes in different parts of the document. If you want to have the behavior well documented you have to write and keep updated a piece of English prose that describes it.

Open Contracting folks are all about making a data standard. Having a piece of code instead of a specification clearly seemed like a wrong approach there. They were already using JSON schema to codify the format of various JSON documents for their procedures. So my idea was to extend the JSON schema format to also encode the information on how to merge consecutive versions of those document.

The result of this line of thought was jsonmerge. For example, to say that arrays appearing in the bar field should be appended instead of replaced, the following schema can be used:

schema = {
            "properties": {
                "bar": {
                    "mergeStrategy": "append"
                }
            }
        }

This way, the definition of the merge process is fairly flexible. jsonmerge contains what I hope are sane defaults for when the strategies are not explicitly defined. This means that the merge operation should not easily break when new fields are added to documents. This kind of schema is also a bit more self-explanatory than a pure Python implementation of the same process. If you already have a JSON schema for your documents, adding merge strategies should be fairly straight-forward.

One more thing that this approach makes possible is that given such an annotated schema for source documents, jsonmerge can automatically produce a JSON schema for the resulting merged document. The merged schema can be used with a schema validator to validate any other implementations of the document merge operation (or as a sanity check to check jsonmerge against itself). Again, this was convenient for Open Contracting since they expect their standards to have multiple implementations.

Since it works on JSON schema documents, the library structure borrows heavily from the jsonschema validator. I believe I managed to make the library general enough so that extending it with additional merge strategies shouldn't be too complicated. The operations performed on the documents are somewhat similar to what version control systems do. Because of that I borrowed terminology from there. jsonmerge documentation and source talks about base and head documents and merge strategies. The meanings are similar to what you would expect from a git man page.

So, if that sounds useful, fetch the latest release from PyPi or get the development version from GitHub. The README should contain further instructions on how to use the library. Consult the docstrings for specific details on the API - there shouldn't be many, as the public interface is fairly limited.

As always, patches and bug reports are welcome.

Posted by Tomaž | Categories: Code | Comments »

On cartoon horses and their lawyers

15.08.2014 19:14

GalaCon is an annual event that is about celebrating pastel colored ponies of all shapes and forms, from animation to traditional art and writing. It's one of the European counterparts to similar events that have popped up on the other side of the Atlantic in recent years. These gatherings were created in the unexpected wake of the amateur creativity that was inspired by Lauren Faust's reimagining of Hasbro's My Little Pony franchise. For the third year in a row GalaCon attracted people from as far away as New Zealand. It's a place where a sizable portion of the attendees wear at least a set of pony ears and a tail, if not a more elaborate equestrian-inspired attire. Needless to say, it can be a somewhat frightful experience at first and definitely not for everyone.

For most people it seems to be a place to get away from the social norms that typically prevent adults from obsessing over stories and imagery meant for audience half their age and often of the opposite gender. While I find the worshiping of creative talents behind the show a bit off-putting, I'm still fascinated by the amateur creations of this community. The artist's booths were a bit high on kitsch ("Incredible. Incredibly expensive" was one comment I overheard), but if you look into the right places on-line, there are still enjoyable and thoughtful stories, art and music to be found.

Meeting people I knew from their creations on the web was a fun experience. However for me this year's GalaCon was also a sobering insight into what kind of a strange mix of creativity, marketing psychology and legal matters goes into creating a cartoon for children these days.

GalaCon and Bronies e.V. flags at Forum am Schlosspark.

A highlight of the event was a panel by M. A. Larson, one of the writers behind the cartoon series. By going step by step through a thread of actual emails exchanged between himself, Lauren Faust and the Hasbro office he demonstrated the process behind creating a script for a single episode.

The exact topic of the panel was not announced beforehand however and all recording of the screen was prohibited, with staff patrolling the aisles to look for cameras. I don't know how much of that was for dramatic effect and how much due to real legal requirements. However even before the panel began that gave a strong impression of the kind of atmosphere a project like this is created in. Especially considering the episode he was discussing aired more then three years ago. I'm sure a lot of people in the audience could quote parts of that script by heart. It has been transcribed, analyzed to the last pixel, remixed and in general picked apart on the Internet years ago.

My Little Pony was called the end of the creator-driven era in animation. So far I thought marketing departments dictated what products should appear on the screen and which characters should be retired to make place for new toy lines. I was surprised to hear that sometimes Hasbro office gets involved even in details like which scene should appear last before the end of an act and the commercial break. That fact was even more surprising since this apparently happened in one of the earliest episodes where the general consensus seems to be that the show was not yet ruined by corporate control over creative talent.

Similar amount of thought seemed to go into possibilities of lawsuits. Larson mentioned their self-censorship of the idea to make characters go paragliding and have them do zip lining instead. Is it really harder to say in a court that some child has been hurt trying to imitate horses sliding along a wire than horses soaring under a parachute?

GalaCon 2014 opening ceremony.

The signs of the absurdity of intellectual property protection these days could also be seen throughout the event. Considering Bronies e.V. paid license fees for the public performance of the show it was ridiculous that they were using low-quality videos from the United States TV broadcasts for projection on the big cinema screen, complete with pop-up advertisements that didn't make sense.

Similarly, the love-hate relationship between copyright holders and non-commercial amateur works is nothing new to report. There were a lot of examples where rabid law firms, tasked with copyright protection and with only tenuous connections back to the mothership, used various extortion tactics to remove remixed content from the web. I still don't understand though what kind of a law justifies cease-and-desist letters for works inspired by unnamed background characters that only appeared for a couple of seconds in the original show.

Evening in front of the Forum am Schlosspark.

In general, GalaCon was a bit more chaotic experience than I would wish for and I left it with mixed feelings. Cartoon ponies on the internet are full of contradictions. While the stories they tell are inspiring and a welcome getaway from daily life, the money-grabs behind them are often depressing. I still believe in the good intentions of these events but the extravagant money throwing at the charity auction made me question a lot of things. With extra fees for faster queues, photos and autographs this year's event felt more like a commercial enterprise than a grassroots community event.

Posted by Tomaž | Categories: Life | Comments »

EuroPython 2014 wrap-up

29.07.2014 14:08

I spent the last week at the EuroPython conference. Since my last visit three years ago, the largest European meeting of the Python community moved from Florence in Italy to the familiar Berlin's Congress Center in Alexanderplatz. Last 7 days were chock-full of talks about everything related to Python, friendly conversations with fellow developers and hacking on this and that odd part of the free software ecosystem. So much so actually that it was a welcome change to spend a day outside, winding down in one of Berlin's numerous parks.

EuroPython 2014 schedule poster in the basement of the BCC.

It was an odd feeling to be back in the BCC. So far I knew it only by the Chaos Communication Congresses it hosted before they moved back to Hamburg. I was half-expecting to see the signature Fairy dust rocket and black Datenpirat flags flying in the front. Instead, part of the front yard was converted to a pleasant outdoor lounge area with an ice cream stand while a bunch of sponsor booths took place of the hackcenter.

Traces of the German hacker scene and the community around the CCC could still be easily seen though. The ubiquitous Club Mate was served on each floor. Blinking All Colors Are Beautiful and Mate Light installations were placed on two floors of the conference center. A few speakers reminisced about the congresses and the CCC Video Operations Center took care of the video recordings with the efficiency we are used to from the Congresses. You could even hear a public request to use more bandwidth now and then.

Constanze Kurz talking about surveillance industry.

There were several talks with a surprisingly political topic for a conference centered around a programming language. I felt odd listening to the opening keynote from Constanze Kurz about one year of Snowden revelations and the involvement of big Internet companies while those same companies were hunting for new employees just outside the lecture hall. Still, I don't think it hurts to remind people that even purely technical work can have consequences in the society.

From the keynotes, it's also worth mentioning Emily Bache and her views on test driven development and the history and future of NumPy development by Travis Oliphant.

Even though I'm not doing as much Python development as I used to in my previous job, there were still plenty of interesting presentations. In the context of scientific computing in Python, I found the talks about the GR plotting framework, simulations with SimPy and probabilistic programming most interesting. Some interesting scientific projects relevant to my work were presented in the poster session as well, although I had problems later finding them on the web to learn more about them. Especially, the work on propagating measurement errors in numerical calculations done by Robert Froeschl at CERN caught my eye.

Park in front of the BCC for EuroPython attendants.

I also attended the matplotlib training, although I was a bit disappointed with it. I was hoping to learn more about advanced usage of this Python library I use daily in my work. I enjoyed the historical examples of data visualization and general advice on how to present graphs in a readable way. However most of the technical details behind plotting and data manipulation weren't new to me and we ran out of time before digging into more advanced topics. I guess it was my fault though, since I now see that the training has a novice tag.

Going away from the topics strictly connected to my work, the talks about automated testing with a simple robot and using Kinect in Python left me with several crazy ideas for potential future projects.

Open Contracting table at EuroPython 2014 sprints.

During the weekend I joined the Open Contracting coding sprint. Jure and I represented our small Open Data group in helping to develop tools for governments to publish information about public tenders and contracts in a standard format. My result from these two days is the jsonmerge library, which deserves a blog post of its own.

In conclusion, I enjoyed EuroPython 2014 immensely. In contrast for instance with 30C3 last year, it was again one of the those events where I got to talk with an unusually large number of people during coffee breaks and social events.

On the other hand, attending it made me realize how much my interests changed after I left Zemanta. I visited Florence during my last days at that job and I was intimately familiar in all nuances of the Python interpreter. There I joined the spring hacking on the CPython distribution. This year however I was more interested in what Python can offer to me as a tool and not that much in developing Python itself.

Posted by Tomaž | Categories: Life | Comments »

Down time

21.07.2014 21:14

You might have noticed that for the past two days or so this website was off-line. The reason for it is a bit curious.

On Friday, 18 July at 17:45, two of my Ethernet switches failed - one in Logatec and one in Ljubljana, separated by around 35 km. They crashed simultaneously at the exactly the same second.

Here are the relevant log entries of two machines connected to them. Both had clocks synchronized via NTP, so the time stamps should be fairly accurate. The machines logged these messages when their Ethernet adapters reported the loss of carrier signal on the cable from their respective switches:

Jul 18 17:45:21 gildedale kernel: [1658547.286187] PHY: sunxi_gmac-0:00 - Link is Down
Jul 18 17:45:21 chandra kernel: [512418.004319] e100 0000:00:0c.0: eth3: NIC Link is Down

These two pieces of equipment were geographically separated and had nothing in common. As far fetched as it seems that they would fail because of a common reason, this appears to be the case here.

Newspapers reported on Friday afternoon that a small explosion occurred at a transformer station in Ljubljana. The distribution company confirms the event, but doesn't share many details. There is also no official source of the exact time, but the first tweet about it appeared at 17:45.

I guess that whatever occurred at the transformer station caused a transient on the power grid that crashed my switches. It must have been fairly short because two other computers that were connected to the same outlet did not reboot or report any problems.

It's curious that this effect threw both of the switches in a state where they didn't reboot, but didn't function either. One of these is a fairly old, low-cost affair that I have seen behave in this way a few times before. The other one is integrated into a Linksys WRT54GL access point that has been fairly reliable so far.

This is the second time this year that something completely unexpected happened with my servers. It's not that I try to run some kind of high-reliability shop, but it is annoying and makes me appreciate the work real system administrators do each day. I guess that's the cost of trying to stay away from the cloud these days.

Posted by Tomaž | Categories: Life | Comments »

Kickstarting failure

16.07.2014 17:37

Yesterday I stumbled upon the list of supposedly life-changing Kickstarter projects that hovered briefly on the front page of Hacker News. While I receive a regular stream of links to more or less feasible crowd-funding projects through various channels of modern communication, this list caught my eye as being particularly full of far fetched, if not down-right fraudulent proposals.

After skimming through a campaign for electric vehicles, written by someone who doesn't know the difference between energy and power, I stopped for a moment on Shawn West's 30-second rechargeable battery. He is asking for $10.000 to build a replacement for ordinary rechargeable batteries using a super capacitor for energy storage instead of an electrochemical cell.

Capacitor, casing and circuit board.

Image by Shawn West

Let's consider for a moment his claims: he says that his patent-pending battery using a lithium-ion super capacitor is roughly equivalent to a typical rechargeable battery. He shows us an AA-sized prototype that supposedly contains two integrated circuits: a voltage regulator and a protective circuit that prevents the capacitor from being over- or under-charged. In the FAQ he mentions that the capacity of his battery is 1150 mAh.

Unsurprisingly, on all of his pictures the capacitor is placed in such a way that the model or capacity rating isn't visible. However, with some image enhancement, it's just possible to read out "YUDEN" on one of the photographs.

Enhanced photograph of Shawn West's capacitor

Taiyo Yuden is in fact a manufacturer of lithium-ion capacitors. Looking through their super capacitor range, there is actually just one model that would fit within the 14 x 50 mm AA sized battery: the 40 farad, 12 x 35 mm cylinder-type LIC1235R3R8406.

Here are its specifications:

C_{cap} = 40 \mathrm{F}
U_{max} = 3.8 \mathrm{V}
U_{min} = 2.2 \mathrm{V}

Let's do some back-of-the envelope calculations: That tiny chip on the circuit board looks like a low-drop linear regulator. In that case, the capacity of the battery given in milliampere-hours is equal to the change in electric charge between the fully charged and fully discharged capacitor (ignoring quiescent current of the regulator):

C_{bat} = \Delta Q = Q_{max} - Q_{min} = C_{cap} \cdot (U_{max} - U_{min})
C_{bat} = 17.8 \mathrm{mAh}

That's barely 1.5% of the claimed capacity!

If we consider for a moment that his circuit actually contains a switching regulator, the situation improves, but only slightly. Given 100% conversion efficiency, the energy that can be extracted from the battery is now equal to the change in electric field energy between the fully charged and fully discharged capacitor:

\Delta W = \frac{C_{cap}}{2}\left(U_{max}^2 - U_{min}^2\right)
\Delta W = 192 \mathrm{J}

Since the inventor claims that his battery does not have a discharge curve, but puts out a steady Ubat = 1.5 V, we can simply convert the energy rating to capacity:

C_{bat} = \frac{\Delta W}{U_{bat}}
C_{bat} = 35.6 \mathrm{mAh}

Obviously, this is much better than the dissipative case above, but the figure is still more than one order of magnitude off from the Kickstarter campaign claim of 1150 mAh. Even giving the author the benefit of doubt and using the largest capacitor from Taiyo Yuden's super capacitor range, the achievable capacity remains much smaller than your vanilla pink-bunny-never-stops alkaline.


Super capacitors are a fascinating component and they certainly have their uses. I kind of like the idea of packaging one into a alkaline battery casing, especially the exposed ring that is used to by-pass the regulator for fast charging. However the claims that this could be used to power your smart phone are ridiculous.

Crowd-funding seems to fuel a big part of the broader-Internet fascination with hardware start-ups these days. I can't help but think that projects with claims that are not challenged in even one of the overly-enthusiastic let's-disrupt-the-industry comments are doing more harm than good to our field.

Posted by Tomaž | Categories: Analog | Comments »

Remembrance

08.07.2014 22:06

Back in my childhood I used to spend a lot of time at my grandparent's cottage. It was a wooden house amid a small orchard, just under the top of a hill outside of Ljubljana. My parents would often drive me there in a red Zastava 101 on weekends and holidays. I played with surplus plumbing parts and old tools that were lying around, slept in the attic and had nightmares from having to go to the outhouse after dark.

The cottage was designed and built by my grandfather as a weekend retreat. He was a mechanical engineer who moved to Ljubljana with my grandmother to one of the first apartment complexes in the city, proudly displaying his professional title above the door bell. He was designing turbines for hydroelectric power plants at the heavy machinery manufacturer Litostroj.

When I wasn't wasting potable water or setting things on fire, one of my favorite ways of spending time there was browsing through a book Elektrotehnika v slikah. I could always find it stashed between two planks above my grandfather's bed. It was bound into a hard crimson cover with gold lettering and even back then already had the smell of an old paperback. The title translates to English as Electrical engineering in pictures and it's actually a Slovenian translation of the German original Elektrotechnik in Bildern by Gustav Büscher.

Elektrotehnika v slikah, cover and page 56

I don't know how that book got there. I doubt that my grandfather learned anything new from it, considering it was printed in 1968 and he finished his 4-book, 5-inch thick thesis on designing Francis turbines 6 years earlier.

In any case, I loved to turn its pages. I looked at the pictures and read the short paragraphs until I knew most of it by heart. First few chapters at least, where the book explains the basics like current, voltage and resistance with simple water analogies. There, almost each paragraph had an illustration that was just serious enough not to appear overly childish and funny enough to keep it interesting. The book was, after all, targeted at adults.

Later chapters went on to describe electric machinery and vacuum tubes with more elaborate mechanical analogies. Judging by the tear-and-wear of the pages, I skipped most of those parts back then. Equations were definitely beyond me at that point, and I'm not sure I even bothered my parents asking about them.

Illustration of HE Završnica from Elektrotehnika v slikah.

In the chapter about hydroelectricity, the Završnica power plant is featured on several occasions. Maybe it was there in the original version of the book. Or, more likely, the illustrations were adapted when the book was translated. Regardless, Završnica was the first public power plant in the region, so it definitely deserved the attention. By the time this book was published, it was already more than 50 years old.

For a long time, if someone would ask me, that plant would be the only one I could name. I didn't know where it was, but the name has definitely stuck in my head.

Surge tank of HE Završnica

Even a month ago I wouldn't know where to put it exactly on the map. I was therefore a bit surprised when I found out that the plant is now being converted into a museum and that I will see it on my visit of HE Moste.

I now know that the dam on river Završnica is a short hike from the town of Žirovnica and is surrounded by a pleasant recreational park. I learned that the pressure pipeline goes under the town and discharges into river Sava below. And I also learned that the prominent building on the top of the hill you see when driving on the Gorenjska highway is its surge tank. Even though machinery in the old turbine hall is being cut open for display, the surge tank above still serves its original purpose. From up close it is an impressive monument to early 20th century hydro engineering.

Mechanical frequency meter in HE Završnica

A walk through the museum revealed many sights that were familiar to me from the book, like the mechanical frequency meter above. I don't think I ever saw a real one before.

HE Završnica museum.

What I also learned on the tour is that HE Moste next door was my grandfather's first big design project at Litostroj.

The original bronze Francis turbines dimensioned by him and cast by Litostroj have long been replaced, damaged by cavitation and sand particles carried from the mountains by river Sava. However one of them is now proudly displayed on a plinth in the park in front of the plant.

After HE Moste, my grandfather went on to design power plants throughout the former Yugoslavia and abroad. His design bureau was involved in projects all over the world. He kept a globe in his office marked with flight paths of his intercontinental flights that circled the Earth.

Bronze Francis turbine on display at HE Moste

The visit to HE Moste and HE Završnica last month has been as much a technical interest for me as it was a way to remember my grandfather. Without doubt he helped spark my interest for engineering. Either by letting me browse his books or watch him spend his days drawing carefully calculated lines on translucent paper.

He died in 2011 and was designing turbines in his home office for as long as he could hold a pen against a drafting table. His cottage is gone as well and Elektrotehnika v slikah is now standing on a bookshelf in my living room. It was nice to see in person one of more than a hundred power plants he helped design during his career that are still producing electricity.

Posted by Tomaž | Categories: Life | Comments »

Visiting HE Moste

22.06.2014 15:09

When you're driving from central Slovenia towards the mountains, the Gorenjska highway crosses the valley of the Sava river. From the viaduct you can see a high dam wedged between sides of a narrow gorge. Behind it is the water reservoir for the Hidroelektrarna Moste.

Viadukt Moste and HE Moste dam.

I've been watching this dam every time I was traveling on that road. For years I wanted to see it up close and the power plant below it. My wish came true when my mum got a contact at the plant and arranged for a guided tour. So my parents and I drove up to Žirovnica last week and visited the facility.

Hidroelektrarna Moste

To my surprise, turbines and the generator hall of the power plant are actually located quite a bit down river from the dam, not directly underneath it as I expected. Water from river Sava is routed through a tunnel under the nearby town Moste, from which the facility also got its name.

What you see above is the steep railway with which heavy machinery has been lowered into the valley during construction.

Safety valve on the high-pressure pipeline for HE Moste

This safety valve on the high-pressure pipe in the tunnel was the first impression of the scale of engineering of this place.

The valve is designed to operate in a fail-safe manner: in case of an emergency, it uses stored potential energy without the need for any external power. The huge red weight shuts off the water supply if electric power is lost or if sensors detect flooding in the facility. Later on in the generator hall we saw emergency buttons similar to fire alarms that would manually trigger it.

Behind the valve is a huge vertical surge tank embedded in rock. Its purpose is to absorb the kinetic energy of the water in the pipeline which could cause damage if the valve would suddenly close.

Even though around 20 m3 of water were flowing per second through the large pipe, there was surprisingly little sound to be heard.

Flotation device near the turbine of HE Moste

The site is actually home to two power plants:

In 1915 Hidroelektrarna Završnica was built here to become the first public power plant in the area. It used water from the river Završnica which was routed from a reservoir higher up in the mountains and discharged into the river Sava.

Later on, river Sava was dammed in 1952 and Hidroelektrarna Moste built alongside the old power plant. Now HE Završnica is no longer in use and is being slowly converted to a museum. Its water supply however has been routed to one of the turbines in HE Moste and is still used intermittently to generate electricity.

Francis turbine in HE Završnica.

This is one of the three 10 MW Francis turbines currently in operation. In contrast to the quiet flow of water in the pipeline, turbine wells were noisy enough that you had to shout to talk to someone.

While we were visiting, one of the generators had to be stopped because of a problem with one of the oil pumps. Our tour was put on hold while repairs were made. After the pump was fixed, the turbine was spun up, generator synchronized with the grid and control of the power plant handed back to the remote operations center in Ljubljana.

Modern hydraulic turbine speed governor.

This is one of the modern hydraulic turbine speed governors that are currently in use in the power plant.

All aspects of the power plant are remotely monitored using a SCADA system. Status of every temperature sensor and pump can be accessed from a central control room. Apart from electrical parameters they also measure and record structural status of the dam and its surroundings. They have automatic strain gauges and a grid of reference points where geometers regularly check for any unexpected earth movements.

In theory the plant could be operated entirely by remote control, but we were told that the company will continue to keep the site staffed round-the-clock. Problems can be detected and solved faster and cheaper if experienced engineers are around. Otherwise a maintenance team would have to be dispatched from somewhere else, which could lead to more downtime or a small problem growing into a more expensive one.

HE Moste, generator 4

In conclusion, I would like to thank the friendly staff of HE Moste for taking the time for us in a busy day and Mr. Pirjevec for guiding us on this tour. We were shown every nook and cranny of the power plant and he gave us a thorough description of the machinery and answered every question we had.

It was a pleasant and informative way to spend a morning and a welcome step away from all the low-power electronics I deal with each day.

Posted by Tomaž | Categories: Life | Comments »

vesna-drivers git visualization

16.06.2014 17:41

Further in the context of two years in development of firmware for VESNA sensor nodes in Logatec, here's another view on the firmware. I used git-big-picture to graph the branching and merging of the source code.

We use a private GitHub repository to collaborate on the code. At the time of writing, it had 16 forks and 7 contributors to the master branch. Before that we used Subversion, but switched to git soon after I joined the team.

vesna-drivers repository branching visualization

Red are commits pointed to by tags, blue are branch heads and white are merge and bifurcation commits. Other uninteresting commits are not shown - zero or more intermediary commits are implied by graph edges. Time, or rather entropy, roughly increases from left to right.

Below is a detail from that picture. If you're curious, you can also download the complete graph as a rather large PDF (most branch names and tags have been redacted though).

vesna-drivers repository branching visualization, detail

At the first glance, this looks pretty chaotic. Definitely our academic environment is a bit less strict regarding the code structure than what you might see in a production project. That's not necessarily a bad thing. Many people here are researchers first and software developers second. For many, this has been their first encounter with source version control.

On the other hand, the whole point of this repository is that work can be shared. Pieces that have been useful in one project are often useful again in another. Searching 16 repositories and countless branches for a driver you might reuse isn't very fun. So some kind of structure is a must.

It's hard to get this balance right tough. On one hand you don't want to be too strict when accepting code to the git master, since then you have less code to share. Often it's even impossible for the reviewer to run-test the code since he might not have the hardware necessary. On the other hand, merging code that will be a maintenance hell later on is counter productive as well. There's no use trying to share code that will cost people more time to debug it than it would to rewrite it from scratch.

We're currently trying to go with a system of GitHub pull requests and a Wiki page that lists all the projects in private branches and I think setting up automated builds was worth the effort. But I guess after two years we're still trying to find the sweet spot.

Posted by Tomaž | Categories: Code | Comments »

Evolution of VESNA firmware size

11.06.2014 21:23

Yesterday I got an idea to plot how the size of the firmware images for VESNA sensor nodes evolved over time. Thanks to my diligent tagging of releases in git and making sure binaries can be built with a single make, all it took was a short bash script to come up with this:

Size of binary firmware image over time.

Only now do I realize that we've been polishing this code base for two years now.

When we originally started working on firmware that would run on spectrum sensing nodes in the Logatec testbed, a decision was made to develop everything from scratch and not go with an existing operating system like Contiki. The rationale was that we don't need all of its facilities and making something from scratch will be simpler than learning and building on existing work.

As it usually happens in such cases, over time we basically developed our own, small operating system. Given all the invested time, it is now hard to make a break from it, even when we have applications that would benefit from a standard platform like Contiki.

Looking at the graph, I'm actually surprised that the size of the firmware didn't increase more than it has. Keep in mind that these binary images are uploaded over a flaky ZigBit network where the best throughput is in the hundreds-of-bytes-per-second range. From the upload times to 50 nodes it certainly felt like it has increased a lot.

I didn't look into what features caused the most size increase. I'm pretty sure the recent large jump between versions 2.40 and 2.42 was because of a new SPI and microSD card driver. Also, one of the size decreases early on was after we added -ffunction-sections and -fdata-sections options to the GCC.

Posted by Tomaž | Categories: Code | Comments »