jsonmerge

20.08.2014 21:01

As I mentioned in my earlier post, my participation at the Open Contracting code sprint during EuroPython resulted in the jsonmerge library. After the conference I slowly cleaned up the remaining few issues and brought up code coverage of unit tests to 99%. The first release is now available from PyPi under the MIT license.

jsonmerge tries to solve a problem that seems simple at first: given a series of structured JSON documents, how to create a single document that contains an aggregate of all their contents. With simple documents that might be as trivial as calling an update() method on a dict:

>>> a = {'foo': 1}
>>> b = {'bar': 2}

>>> c = a.update(b)
>>> c
{'foo': 1, 'bar': 2}

However, even with just two plain dictionaries, things can quickly get complicated. What should happen if both documents contain a field with the same name? Should a later value overwrite the earlier one? Or should the resulting document have in that place a list that contains both values? Source JSON documents themselves can also contain arrays (or arrays of arrays) and handling those is even less straightforward than dictionaries in this example.

Often I've seen a problem like this solved in application code - it's relatively simple to encode your wishes in several hundreds lines of Python. However JSON is a very flexible format while such code is typically brittle. Change the input document a bit and more often than not your code will start throwing KeyErrors left and right. Another problem with this approach is that it's often not obvious from the code what kind of a strategy is taken for merging changes in different parts of the document. If you want to have the behavior well documented you have to write and keep updated a piece of English prose that describes it.

Open Contracting folks are all about making a data standard. Having a piece of code instead of a specification clearly seemed like a wrong approach there. They were already using JSON schema to codify the format of various JSON documents for their procedures. So my idea was to extend the JSON schema format to also encode the information on how to merge consecutive versions of those document.

The result of this line of thought was jsonmerge. For example, to say that arrays appearing in the bar field should be appended instead of replaced, the following schema can be used:

schema = {
            "properties": {
                "bar": {
                    "mergeStrategy": "append"
                }
            }
        }

This way, the definition of the merge process is fairly flexible. jsonmerge contains what I hope are sane defaults for when the strategies are not explicitly defined. This means that the merge operation should not easily break when new fields are added to documents. This kind of schema is also a bit more self-explanatory than a pure Python implementation of the same process. If you already have a JSON schema for your documents, adding merge strategies should be fairly straight-forward.

One more thing that this approach makes possible is that given such an annotated schema for source documents, jsonmerge can automatically produce a JSON schema for the resulting merged document. The merged schema can be used with a schema validator to validate any other implementations of the document merge operation (or as a sanity check to check jsonmerge against itself). Again, this was convenient for Open Contracting since they expect their standards to have multiple implementations.

Since it works on JSON schema documents, the library structure borrows heavily from the jsonschema validator. I believe I managed to make the library general enough so that extending it with additional merge strategies shouldn't be too complicated. The operations performed on the documents are somewhat similar to what version control systems do. Because of that I borrowed terminology from there. jsonmerge documentation and source talks about base and head documents and merge strategies. The meanings are similar to what you would expect from a git man page.

So, if that sounds useful, fetch the latest release from PyPi or get the development version from GitHub. The README should contain further instructions on how to use the library. Consult the docstrings for specific details on the API - there shouldn't be many, as the public interface is fairly limited.

As always, patches and bug reports are welcome.

Posted by Tomaž | Categories: Code | Comments »

On cartoon horses and their lawyers

15.08.2014 19:14

GalaCon is an annual event that is about celebrating pastel colored ponies of all shapes and forms, from animation to traditional art and writing. It's one of the European counterparts to similar events that have popped up on the other side of the Atlantic in recent years. These gatherings were created in the unexpected wake of the amateur creativity that was inspired by Lauren Faust's reimagining of Hasbro's My Little Pony franchise. For the third year in a row GalaCon attracted people from as far away as New Zealand. It's a place where a sizable portion of the attendees wear at least a set of pony ears and a tail, if not a more elaborate equestrian-inspired attire. Needless to say, it can be a somewhat frightful experience at first and definitely not for everyone.

For most people it seems to be a place to get away from the social norms that typically prevent adults from obsessing over stories and imagery meant for audience half their age and often of the opposite gender. While I find the worshiping of creative talents behind the show a bit off-putting, I'm still fascinated by the amateur creations of this community. The artist's booths were a bit high on kitsch ("Incredible. Incredibly expensive" was one comment I overheard), but if you look into the right places on-line, there are still enjoyable and thoughtful stories, art and music to be found.

Meeting people I knew from their creations on the web was a fun experience. However for me this year's GalaCon was also a sobering insight into what kind of a strange mix of creativity, marketing psychology and legal matters goes into creating a cartoon for children these days.

GalaCon and Bronies e.V. flags at Forum am Schlosspark.

A highlight of the event was a panel by M. A. Larson, one of the writers behind the cartoon series. By going step by step through a thread of actual emails exchanged between himself, Lauren Faust and the Hasbro office he demonstrated the process behind creating a script for a single episode.

The exact topic of the panel was not announced beforehand however and all recording of the screen was prohibited, with staff patrolling the aisles to look for cameras. I don't know how much of that was for dramatic effect and how much due to real legal requirements. However even before the panel began that gave a strong impression of the kind of atmosphere a project like this is created in. Especially considering the episode he was discussing aired more then three years ago. I'm sure a lot of people in the audience could quote parts of that script by heart. It has been transcribed, analyzed to the last pixel, remixed and in general picked apart on the Internet years ago.

My Little Pony was called the end of the creator-driven era in animation. So far I thought marketing departments dictated what products should appear on the screen and which characters should be retired to make place for new toy lines. I was surprised to hear that sometimes Hasbro office gets involved even in details like which scene should appear last before the end of an act and the commercial break. That fact was even more surprising since this apparently happened in one of the earliest episodes where the general consensus seems to be that the show was not yet ruined by corporate control over creative talent.

Similar amount of thought seemed to go into possibilities of lawsuits. Larson mentioned their self-censorship of the idea to make characters go paragliding and have them do zip lining instead. Is it really harder to say in a court that some child has been hurt trying to imitate horses sliding along a wire than horses soaring under a parachute?

GalaCon 2014 opening ceremony.

The signs of the absurdity of intellectual property protection these days could also be seen throughout the event. Considering Bronies e.V. paid license fees for the public performance of the show it was ridiculous that they were using low-quality videos from the United States TV broadcasts for projection on the big cinema screen, complete with pop-up advertisements that didn't make sense.

Similarly, the love-hate relationship between copyright holders and non-commercial amateur works is nothing new to report. There were a lot of examples where rabid law firms, tasked with copyright protection and with only tenuous connections back to the mothership, used various extortion tactics to remove remixed content from the web. I still don't understand though what kind of a law justifies cease-and-desist letters for works inspired by unnamed background characters that only appeared for a couple of seconds in the original show.

Evening in front of the Forum am Schlosspark.

In general, GalaCon was a bit more chaotic experience than I would wish for and I left it with mixed feelings. Cartoon ponies on the internet are full of contradictions. While the stories they tell are inspiring and a welcome getaway from daily life, the money-grabs behind them are often depressing. I still believe in the good intentions of these events but the extravagant money throwing at the charity auction made me question a lot of things. With extra fees for faster queues, photos and autographs this year's event felt more like a commercial enterprise than a grassroots community event.

Posted by Tomaž | Categories: Life | Comments »

EuroPython 2014 wrap-up

29.07.2014 14:08

I spent the last week at the EuroPython conference. Since my last visit three years ago, the largest European meeting of the Python community moved from Florence in Italy to the familiar Berlin's Congress Center in Alexanderplatz. Last 7 days were chock-full of talks about everything related to Python, friendly conversations with fellow developers and hacking on this and that odd part of the free software ecosystem. So much so actually that it was a welcome change to spend a day outside, winding down in one of Berlin's numerous parks.

EuroPython 2014 schedule poster in the basement of the BCC.

It was an odd feeling to be back in the BCC. So far I knew it only by the Chaos Communication Congresses it hosted before they moved back to Hamburg. I was half-expecting to see the signature Fairy dust rocket and black Datenpirat flags flying in the front. Instead, part of the front yard was converted to a pleasant outdoor lounge area with an ice cream stand while a bunch of sponsor booths took place of the hackcenter.

Traces of the German hacker scene and the community around the CCC could still be easily seen though. The ubiquitous Club Mate was served on each floor. Blinking All Colors Are Beautiful and Mate Light installations were placed on two floors of the conference center. A few speakers reminisced about the congresses and the CCC Video Operations Center took care of the video recordings with the efficiency we are used to from the Congresses. You could even hear a public request to use more bandwidth now and then.

Constanze Kurz talking about surveillance industry.

There were several talks with a surprisingly political topic for a conference centered around a programming language. I felt odd listening to the opening keynote from Constanze Kurz about one year of Snowden revelations and the involvement of big Internet companies while those same companies were hunting for new employees just outside the lecture hall. Still, I don't think it hurts to remind people that even purely technical work can have consequences in the society.

From the keynotes, it's also worth mentioning Emily Bache and her views on test driven development and the history and future of NumPy development by Travis Oliphant.

Even though I'm not doing as much Python development as I used to in my previous job, there were still plenty of interesting presentations. In the context of scientific computing in Python, I found the talks about the GR plotting framework, simulations with SimPy and probabilistic programming most interesting. Some interesting scientific projects relevant to my work were presented in the poster session as well, although I had problems later finding them on the web to learn more about them. Especially, the work on propagating measurement errors in numerical calculations done by Robert Froeschl at CERN caught my eye.

Park in front of the BCC for EuroPython attendants.

I also attended the matplotlib training, although I was a bit disappointed with it. I was hoping to learn more about advanced usage of this Python library I use daily in my work. I enjoyed the historical examples of data visualization and general advice on how to present graphs in a readable way. However most of the technical details behind plotting and data manipulation weren't new to me and we ran out of time before digging into more advanced topics. I guess it was my fault though, since I now see that the training has a novice tag.

Going away from the topics strictly connected to my work, the talks about automated testing with a simple robot and using Kinect in Python left me with several crazy ideas for potential future projects.

Open Contracting table at EuroPython 2014 sprints.

During the weekend I joined the Open Contracting coding sprint. Jure and I represented our small Open Data group in helping to develop tools for governments to publish information about public tenders and contracts in a standard format. My result from these two days is the jsonmerge library, which deserves a blog post of its own.

In conclusion, I enjoyed EuroPython 2014 immensely. In contrast for instance with 30C3 last year, it was again one of the those events where I got to talk with an unusually large number of people during coffee breaks and social events.

On the other hand, attending it made me realize how much my interests changed after I left Zemanta. I visited Florence during my last days at that job and I was intimately familiar in all nuances of the Python interpreter. There I joined the spring hacking on the CPython distribution. This year however I was more interested in what Python can offer to me as a tool and not that much in developing Python itself.

Posted by Tomaž | Categories: Life | Comments »

Down time

21.07.2014 21:14

You might have noticed that for the past two days or so this website was off-line. The reason for it is a bit curious.

On Friday, 18 July at 17:45, two of my Ethernet switches failed - one in Logatec and one in Ljubljana, separated by around 35 km. They crashed simultaneously at the exactly the same second.

Here are the relevant log entries of two machines connected to them. Both had clocks synchronized via NTP, so the time stamps should be fairly accurate. The machines logged these messages when their Ethernet adapters reported the loss of carrier signal on the cable from their respective switches:

Jul 18 17:45:21 gildedale kernel: [1658547.286187] PHY: sunxi_gmac-0:00 - Link is Down
Jul 18 17:45:21 chandra kernel: [512418.004319] e100 0000:00:0c.0: eth3: NIC Link is Down

These two pieces of equipment were geographically separated and had nothing in common. As far fetched as it seems that they would fail because of a common reason, this appears to be the case here.

Newspapers reported on Friday afternoon that a small explosion occurred at a transformer station in Ljubljana. The distribution company confirms the event, but doesn't share many details. There is also no official source of the exact time, but the first tweet about it appeared at 17:45.

I guess that whatever occurred at the transformer station caused a transient on the power grid that crashed my switches. It must have been fairly short because two other computers that were connected to the same outlet did not reboot or report any problems.

It's curious that this effect threw both of the switches in a state where they didn't reboot, but didn't function either. One of these is a fairly old, low-cost affair that I have seen behave in this way a few times before. The other one is integrated into a Linksys WRT54GL access point that has been fairly reliable so far.

This is the second time this year that something completely unexpected happened with my servers. It's not that I try to run some kind of high-reliability shop, but it is annoying and makes me appreciate the work real system administrators do each day. I guess that's the cost of trying to stay away from the cloud these days.

Posted by Tomaž | Categories: Life | Comments »

Kickstarting failure

16.07.2014 17:37

Yesterday I stumbled upon the list of supposedly life-changing Kickstarter projects that hovered briefly on the front page of Hacker News. While I receive a regular stream of links to more or less feasible crowd-funding projects through various channels of modern communication, this list caught my eye as being particularly full of far fetched, if not down-right fraudulent proposals.

After skimming through a campaign for electric vehicles, written by someone who doesn't know the difference between energy and power, I stopped for a moment on Shawn West's 30-second rechargeable battery. He is asking for $10.000 to build a replacement for ordinary rechargeable batteries using a super capacitor for energy storage instead of an electrochemical cell.

Capacitor, casing and circuit board.

Image by Shawn West

Let's consider for a moment his claims: he says that his patent-pending battery using a lithium-ion super capacitor is roughly equivalent to a typical rechargeable battery. He shows us an AA-sized prototype that supposedly contains two integrated circuits: a voltage regulator and a protective circuit that prevents the capacitor from being over- or under-charged. In the FAQ he mentions that the capacity of his battery is 1150 mAh.

Unsurprisingly, on all of his pictures the capacitor is placed in such a way that the model or capacity rating isn't visible. However, with some image enhancement, it's just possible to read out "YUDEN" on one of the photographs.

Enhanced photograph of Shawn West's capacitor

Taiyo Yuden is in fact a manufacturer of lithium-ion capacitors. Looking through their super capacitor range, there is actually just one model that would fit within the 14 x 50 mm AA sized battery: the 40 farad, 12 x 35 mm cylinder-type LIC1235R3R8406.

Here are its specifications:

C_{cap} = 40 \mathrm{F}
U_{max} = 3.8 \mathrm{V}
U_{min} = 2.2 \mathrm{V}

Let's do some back-of-the envelope calculations: That tiny chip on the circuit board looks like a low-drop linear regulator. In that case, the capacity of the battery given in milliampere-hours is equal to the change in electric charge between the fully charged and fully discharged capacitor (ignoring quiescent current of the regulator):

C_{bat} = \Delta Q = Q_{max} - Q_{min} = C_{cap} \cdot (U_{max} - U_{min})
C_{bat} = 17.8 \mathrm{mAh}

That's barely 1.5% of the claimed capacity!

If we consider for a moment that his circuit actually contains a switching regulator, the situation improves, but only slightly. Given 100% conversion efficiency, the energy that can be extracted from the battery is now equal to the change in electric field energy between the fully charged and fully discharged capacitor:

\Delta W = \frac{C_{cap}}{2}\left(U_{max}^2 - U_{min}^2\right)
\Delta W = 192 \mathrm{J}

Since the inventor claims that his battery does not have a discharge curve, but puts out a steady Ubat = 1.5 V, we can simply convert the energy rating to capacity:

C_{bat} = \frac{\Delta W}{U_{bat}}
C_{bat} = 35.6 \mathrm{mAh}

Obviously, this is much better than the dissipative case above, but the figure is still more than one order of magnitude off from the Kickstarter campaign claim of 1150 mAh. Even giving the author the benefit of doubt and using the largest capacitor from Taiyo Yuden's super capacitor range, the achievable capacity remains much smaller than your vanilla pink-bunny-never-stops alkaline.


Super capacitors are a fascinating component and they certainly have their uses. I kind of like the idea of packaging one into a alkaline battery casing, especially the exposed ring that is used to by-pass the regulator for fast charging. However the claims that this could be used to power your smart phone are ridiculous.

Crowd-funding seems to fuel a big part of the broader-Internet fascination with hardware start-ups these days. I can't help but think that projects with claims that are not challenged in even one of the overly-enthusiastic let's-disrupt-the-industry comments are doing more harm than good to our field.

Posted by Tomaž | Categories: Analog | Comments »

Remembrance

08.07.2014 22:06

Back in my childhood I used to spend a lot of time at my grandparent's cottage. It was a wooden house amid a small orchard, just under the top of a hill outside of Ljubljana. My parents would often drive me there in a red Zastava 101 on weekends and holidays. I played with surplus plumbing parts and old tools that were lying around, slept in the attic and had nightmares from having to go to the outhouse after dark.

The cottage was designed and built by my grandfather as a weekend retreat. He was a mechanical engineer who moved to Ljubljana with my grandmother to one of the first apartment complexes in the city, proudly displaying his professional title above the door bell. He was designing turbines for hydroelectric power plants at the heavy machinery manufacturer Litostroj.

When I wasn't wasting potable water or setting things on fire, one of my favorite ways of spending time there was browsing through a book Elektrotehnika v slikah. I could always find it stashed between two planks above my grandfather's bed. It was bound into a hard crimson cover with gold lettering and even back then already had the smell of an old paperback. The title translates to English as Electrical engineering in pictures and it's actually a Slovenian translation of the German original Elektrotechnik in Bildern by Gustav Büscher.

Elektrotehnika v slikah, cover and page 56

I don't know how that book got there. I doubt that my grandfather learned anything new from it, considering it was printed in 1968 and he finished his 4-book, 5-inch thick thesis on designing Francis turbines 6 years earlier.

In any case, I loved to turn its pages. I looked at the pictures and read the short paragraphs until I knew most of it by heart. First few chapters at least, where the book explains the basics like current, voltage and resistance with simple water analogies. There, almost each paragraph had an illustration that was just serious enough not to appear overly childish and funny enough to keep it interesting. The book was, after all, targeted at adults.

Later chapters went on to describe electric machinery and vacuum tubes with more elaborate mechanical analogies. Judging by the tear-and-wear of the pages, I skipped most of those parts back then. Equations were definitely beyond me at that point, and I'm not sure I even bothered my parents asking about them.

Illustration of HE Završnica from Elektrotehnika v slikah.

In the chapter about hydroelectricity, the Završnica power plant is featured on several occasions. Maybe it was there in the original version of the book. Or, more likely, the illustrations were adapted when the book was translated. Regardless, Završnica was the first public power plant in the region, so it definitely deserved the attention. By the time this book was published, it was already more than 50 years old.

For a long time, if someone would ask me, that plant would be the only one I could name. I didn't know where it was, but the name has definitely stuck in my head.

Surge tank of HE Završnica

Even a month ago I wouldn't know where to put it exactly on the map. I was therefore a bit surprised when I found out that the plant is now being converted into a museum and that I will see it on my visit of HE Moste.

I now know that the dam on river Završnica is a short hike from the town of Žirovnica and is surrounded by a pleasant recreational park. I learned that the pressure pipeline goes under the town and discharges into river Sava below. And I also learned that the prominent building on the top of the hill you see when driving on the Gorenjska highway is its surge tank. Even though machinery in the old turbine hall is being cut open for display, the surge tank above still serves its original purpose. From up close it is an impressive monument to early 20th century hydro engineering.

Mechanical frequency meter in HE Završnica

A walk through the museum revealed many sights that were familiar to me from the book, like the mechanical frequency meter above. I don't think I ever saw a real one before.

HE Završnica museum.

What I also learned on the tour is that HE Moste next door was my grandfather's first big design project at Litostroj.

The original bronze Francis turbines dimensioned by him and cast by Litostroj have long been replaced, damaged by cavitation and sand particles carried from the mountains by river Sava. However one of them is now proudly displayed on a plinth in the park in front of the plant.

After HE Moste, my grandfather went on to design power plants throughout the former Yugoslavia and abroad. His design bureau was involved in projects all over the world. He kept a globe in his office marked with flight paths of his intercontinental flights that circled the Earth.

Bronze Francis turbine on display at HE Moste

The visit to HE Moste and HE Završnica last month has been as much a technical interest for me as it was a way to remember my grandfather. Without doubt he helped spark my interest for engineering. Either by letting me browse his books or watch him spend his days drawing carefully calculated lines on translucent paper.

He died in 2011 and was designing turbines in his home office for as long as he could hold a pen against a drafting table. His cottage is gone as well and Elektrotehnika v slikah is now standing on a bookshelf in my living room. It was nice to see in person one of more than a hundred power plants he helped design during his career that are still producing electricity.

Posted by Tomaž | Categories: Life | Comments »

Visiting HE Moste

22.06.2014 15:09

When you're driving from central Slovenia towards the mountains, the Gorenjska highway crosses the valley of the Sava river. From the viaduct you can see a high dam wedged between sides of a narrow gorge. Behind it is the water reservoir for the Hidroelektrarna Moste.

Viadukt Moste and HE Moste dam.

I've been watching this dam every time I was traveling on that road. For years I wanted to see it up close and the power plant below it. My wish came true when my mum got a contact at the plant and arranged for a guided tour. So my parents and I drove up to Žirovnica last week and visited the facility.

Hidroelektrarna Moste

To my surprise, turbines and the generator hall of the power plant are actually located quite a bit down river from the dam, not directly underneath it as I expected. Water from river Sava is routed through a tunnel under the nearby town Moste, from which the facility also got its name.

What you see above is the steep railway with which heavy machinery has been lowered into the valley during construction.

Safety valve on the high-pressure pipeline for HE Moste

This safety valve on the high-pressure pipe in the tunnel was the first impression of the scale of engineering of this place.

The valve is designed to operate in a fail-safe manner: in case of an emergency, it uses stored potential energy without the need for any external power. The huge red weight shuts off the water supply if electric power is lost or if sensors detect flooding in the facility. Later on in the generator hall we saw emergency buttons similar to fire alarms that would manually trigger it.

Behind the valve is a huge vertical surge tank embedded in rock. Its purpose is to absorb the kinetic energy of the water in the pipeline which could cause damage if the valve would suddenly close.

Even though around 20 m3 of water were flowing per second through the large pipe, there was surprisingly little sound to be heard.

Flotation device near the turbine of HE Moste

The site is actually home to two power plants:

In 1915 Hidroelektrarna Završnica was built here to become the first public power plant in the area. It used water from the river Završnica which was routed from a reservoir higher up in the mountains and discharged into the river Sava.

Later on, river Sava was dammed in 1952 and Hidroelektrarna Moste built alongside the old power plant. Now HE Završnica is no longer in use and is being slowly converted to a museum. Its water supply however has been routed to one of the turbines in HE Moste and is still used intermittently to generate electricity.

Francis turbine in HE Završnica.

This is one of the three 10 MW Francis turbines currently in operation. In contrast to the quiet flow of water in the pipeline, turbine wells were noisy enough that you had to shout to talk to someone.

While we were visiting, one of the generators had to be stopped because of a problem with one of the oil pumps. Our tour was put on hold while repairs were made. After the pump was fixed, the turbine was spun up, generator synchronized with the grid and control of the power plant handed back to the remote operations center in Ljubljana.

Modern hydraulic turbine speed governor.

This is one of the modern hydraulic turbine speed governors that are currently in use in the power plant.

All aspects of the power plant are remotely monitored using a SCADA system. Status of every temperature sensor and pump can be accessed from a central control room. Apart from electrical parameters they also measure and record structural status of the dam and its surroundings. They have automatic strain gauges and a grid of reference points where geometers regularly check for any unexpected earth movements.

In theory the plant could be operated entirely by remote control, but we were told that the company will continue to keep the site staffed round-the-clock. Problems can be detected and solved faster and cheaper if experienced engineers are around. Otherwise a maintenance team would have to be dispatched from somewhere else, which could lead to more downtime or a small problem growing into a more expensive one.

HE Moste, generator 4

In conclusion, I would like to thank the friendly staff of HE Moste for taking the time for us in a busy day and Mr. Pirjevec for guiding us on this tour. We were shown every nook and cranny of the power plant and he gave us a thorough description of the machinery and answered every question we had.

It was a pleasant and informative way to spend a morning and a welcome step away from all the low-power electronics I deal with each day.

Posted by Tomaž | Categories: Life | Comments »

vesna-drivers git visualization

16.06.2014 17:41

Further in the context of two years in development of firmware for VESNA sensor nodes in Logatec, here's another view on the firmware. I used git-big-picture to graph the branching and merging of the source code.

We use a private GitHub repository to collaborate on the code. At the time of writing, it had 16 forks and 7 contributors to the master branch. Before that we used Subversion, but switched to git soon after I joined the team.

vesna-drivers repository branching visualization

Red are commits pointed to by tags, blue are branch heads and white are merge and bifurcation commits. Other uninteresting commits are not shown - zero or more intermediary commits are implied by graph edges. Time, or rather entropy, roughly increases from left to right.

Below is a detail from that picture. If you're curious, you can also download the complete graph as a rather large PDF (most branch names and tags have been redacted though).

vesna-drivers repository branching visualization, detail

At the first glance, this looks pretty chaotic. Definitely our academic environment is a bit less strict regarding the code structure than what you might see in a production project. That's not necessarily a bad thing. Many people here are researchers first and software developers second. For many, this has been their first encounter with source version control.

On the other hand, the whole point of this repository is that work can be shared. Pieces that have been useful in one project are often useful again in another. Searching 16 repositories and countless branches for a driver you might reuse isn't very fun. So some kind of structure is a must.

It's hard to get this balance right tough. On one hand you don't want to be too strict when accepting code to the git master, since then you have less code to share. Often it's even impossible for the reviewer to run-test the code since he might not have the hardware necessary. On the other hand, merging code that will be a maintenance hell later on is counter productive as well. There's no use trying to share code that will cost people more time to debug it than it would to rewrite it from scratch.

We're currently trying to go with a system of GitHub pull requests and a Wiki page that lists all the projects in private branches and I think setting up automated builds was worth the effort. But I guess after two years we're still trying to find the sweet spot.

Posted by Tomaž | Categories: Code | Comments »

Evolution of VESNA firmware size

11.06.2014 21:23

Yesterday I got an idea to plot how the size of the firmware images for VESNA sensor nodes evolved over time. Thanks to my diligent tagging of releases in git and making sure binaries can be built with a single make, all it took was a short bash script to come up with this:

Size of binary firmware image over time.

Only now do I realize that we've been polishing this code base for two years now.

When we originally started working on firmware that would run on spectrum sensing nodes in the Logatec testbed, a decision was made to develop everything from scratch and not go with an existing operating system like Contiki. The rationale was that we don't need all of its facilities and making something from scratch will be simpler than learning and building on existing work.

As it usually happens in such cases, over time we basically developed our own, small operating system. Given all the invested time, it is now hard to make a break from it, even when we have applications that would benefit from a standard platform like Contiki.

Looking at the graph, I'm actually surprised that the size of the firmware didn't increase more than it has. Keep in mind that these binary images are uploaded over a flaky ZigBit network where the best throughput is in the hundreds-of-bytes-per-second range. From the upload times to 50 nodes it certainly felt like it has increased a lot.

I didn't look into what features caused the most size increase. I'm pretty sure the recent large jump between versions 2.40 and 2.42 was because of a new SPI and microSD card driver. Also, one of the size decreases early on was after we added -ffunction-sections and -fdata-sections options to the GCC.

Posted by Tomaž | Categories: Code | Comments »

Testing VESNA's power supply

08.06.2014 18:22

Now that I have Re:load, I seem to end up checking every power supply that lands on my desk if it meets its specifications. I put VESNA's to the test yesterday.

VESNA core board has a switching power supply that supplies around 5V and 3.3V to any expansion boards connected to it. The source can be a battery, solar cell, USB cable or a DC voltage from some external supply. I tested it with a 12V external supply, which is the most common case when you power a node from the grid using a wall wart or something like that.

Testing VESNA SNC source current capabilities on expansion connector.

VCC pin on the expansion connector supplies 3.3V. This line also powers VESNA's main ARM CPU. Datasheet says it can supply at most 500 mA to the expansion board and this is approximately what I've also seen in my test. With a setting on Re:load higher than 500 mA, the voltage fell sharply to around 2 V where Re:load wasn't acting as a current source anymore.

I've measured the voltage using my oscilloscope, so the resolution on the vertical axis isn't as good as it could be. Still, I think 40 mΩ is a fair estimate of the internal resistance for VCC.

VCC voltage versus output current.

Voltage on the VPP pin is typically around 4.5 V. This pin is also supposed to supply a maximum of 500 mA to the expansion board. In my test it managed just a little bit below that, but considering the accuracy of my measurements that's within the margin of error.

VPP voltage versus output current.

I also checked the ripple on both VCC and VPP pins. With low loads the power supply seems to operate in discontinuous mode and the ripple is a bit high at 100 mV peak-to-peak. With higher loads, it goes into continuous mode and the peak-to-peak voltage falls. This is how it looks like on the VPP pin with near maximum load:

VPP voltage ripple when under load.

So, I'm happy to say that everything looks here as it should. The ripple is a bit high for my taste, especially when powering analog electronics.

Part of why I tested this is because my new UHF receiver will put a higher load on this power supply than other expansions I've worked with on VESNA. With around 350 mA of load my board puts on the VPP line, it looks like VESNA will be able to power it without problems. It won't be able to power two receivers at once however and some auxiliary power supply will be necessary for that.

Posted by Tomaž | Categories: Analog | Comments »