Avian’s Blog

I recently bought a new USB flash drive. By chance I noticed that it didn't appear to be empty. The drive came pre-formatted with a VFAT file system that indeed did not contain any files when I took the drive out of the box. However the raw contents of the drive seemed to contain some data that was apparently written to it in the factory. Since I'm used to seeing just zeros or 0xff on new devices I was curious what was on it and whether it contains any hints on how these are tested in the factory.

This is the flash drive in question. It's an ADATA S102 Pro 32 GB flash drive.

usb 9-3.3: New USB device found, idVendor=125f, idProduct=312b, bcdDevice=11.00
usb 9-3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 9-3.3: Product: ADATA USB Flash Drive
usb 9-3.3: Manufacturer: ADATA

This first thing I did was to dump an image of the drive in factory condition to a 32 GB file on my hard drive. I then used the binvis tool from Aldo Cortesi to visualize its contents.

The following picture shows just the first 32 MiB using the zig-zag curve. Beginning of the drive is in the top left corner. Offset increases from left to right and top to bottom:

There's mostly zeros in the first 16 MiB of the drive, with a few small isolated chunks of data and strings. At very the start there's the default partition table and the empty FAT. One of the other non-zero chunks has some recognizable strings referring to a burn-in test.

0101c000  41 44 41 54 41 20 55 46  44 20 20 08 00 00 00 00  |ADATA UFD  .....|
0101c010  00 00 00 00 00 00 56 5a  7e 33 00 00 00 00 00 00  |......VZ~3......|
0101c020  e5 73 00 74 00 20 00 66  00 69 00 0f 00 b5 6c 00  |.s.t. .f.i....l.|
0101c030  65 00 73 00 00 00 ff ff  ff ff 00 00 ff ff ff ff  |e.s.............|
0101c040  e5 42 00 75 00 72 00 6e  00 49 00 0f 00 b5 6e 00  |.B.u.r.n.I....n.|
0101c050  54 00 65 00 73 00 74 00  20 00 00 00 74 00 65 00  |T.e.s.t. ...t.e.|
0101c060  e5 55 52 4e 49 4e 7e 31  20 20 20 10 00 7b 83 be  |.URNIN~1   ..{..|
0101c070  74 52 74 52 00 00 84 be  74 52 03 00 00 40 00 00  |tRtR....tR...@..|
0101c080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Other than that I didn't find anything interesting. This seems to be just a remnant of some filesystem metadata. Chunks are too small to contain actual test code and I didn't expect that to be on the drive anyway.

Following the first 16 MiB is a pseudo-random pattern. It starts at offset 0x1024000 and repeats every 4 MiB (4194304 bytes). Extracting just a single repetition of the pattern and plotting it with binvis reveals that it has some internal structure to it. This plot is using the gradient color map and the same zig-zag curve as before:

I also played around with different ways of distributing the pattern onto a grid using matplotlib. This is how it looks like when drawn onto a 2048 by 2048 grid. The plot is zoomed into just one corner of the pattern:

I don't know the reason behind this pattern. It might just be some pseudo-random algorithm they are using and the values themselves don't matter. I don't recognize any obvious bit test patterns in it. It seems that on every 2048 byte line the byte values in the pattern increase using a different step. Histogram of byte values shows that all 256 values are represented approximately equally.

Interestingly, the test pattern does not continue throughout the entire 32 GB of the drive. It stops at approximately 3 GB. The drive just contains zeros after that. Does this mean that only the first 3 GB has been factory tested? Or maybe there was a bug in the process and only this first chunk didn't get zeroed out after the test while the rest of the drive did?

A couple of months ago I was writing about 3D-printable replacement drive rails for my Chieftec PC enclosure. Back then I've designed and printed some parts that were functional enough to use for mounting a new set of 3.5" hard drives into my computer and I have been using them since. However I was bothered by the fact that the new rails required two parts to be glued together. I've now updated the design so that the latch snaps onto the base part of the rail. It's purely a friction fit and hence assembly of the new rails doesn't require any glue.

Having to glue together 12 rails for the complete set of drives was bothersome. However the main reason why I wanted to avoid using glue was because I couldn't find one that would bond well to the PETG plastic that my parts were printed from. Every glue I tried produced a very weak bond that soon fell apart. I tried to modify the design so that it minimized the stress on the bond, but that didn't really work. I've heard that this specific glue produces good results with the filament I used, however I could not find a shop that would have it in stock.

Coming up with a design where the latch just snaps into the base required two more round trips between CAD and the 3D printer. I find that the most troublesome part of any 3D printed design is always the place where two parts need to slide or engage with each other. Each printer has slightly different tolerances. Often this differs even between prints on the same printer. It's hard to find the exact amount of space you need to leave in the STL files between surfaces. Too much and the parts fit too loosely, too little and the parts don't fit together or break when they are assembled.

I've written in one of my earlier posts that I was also worried about the 3D printed rails getting soft when in contact with the warm hard drives. I've been using them now for close to 2 months and so far I haven't seen any signs of deformation due to heat. On the other hand, the winter has barely ended. I expect the drives to reach higher temperatures in summer.

I've put the new designs in place of the old ones. There's now also a README file there that has some condensed instructions for printing.

In the end, this took way more time than I anticipated. 3D printers are fun and convenient, but getting to a design that works well can still be very time consuming. In this particular case it was also mostly due to me being stubborn and wanting a replacement that functions more or less exactly like the original. It turned out that even without the latch the rails function quite well. There are sheet metal springs in the case that grab onto the holes for the screws on the rails. At least in my enclosure, these springs alone provide enough friction that drives are held pretty well even without the additional security of the latch action on the rail itself.

I have been preoccupied with data storage issues lately. What I though would be a simple installation of a solid state drive into my desktop PC turned into a month-long project. I found out I need to design and make new drive rails, decided to do some overdue restructuring of the RAID array and then had to replace two failing drives. In any case, a thing that caught my eye recently was a warning about RAID 5 I saw repeated on a few places on the web:

Note: RAID 5 is a common choice due to its combination of speed and data redundancy. The caveat is that if one drive were to fail and another drive failed before that drive was replaced, all data will be lost. Furthermore, with modern disk sizes and expected unrecoverable read error (URE) rates on consumer disks, the rebuild of a 4TiB array is expected (i.e. higher than 50% chance) to have at least one URE. Because of this, RAID 5 is no longer advised by the storage industry.

This is how I understand the second part of the warning: it talks about a RAID 5 array with total usable capacity of 4 TiB. Such an array would typically consist of three 2-terabyte disks. In the described scenario one of the disks has failed and, after adding in a replacement drive, the array is restored by reading the contents of the remaining two disks. This means we need to read out 2 times 2 terabytes of data without errors to successfully restore the array.

I was surprised by the stated higher-than-50% chance of a read error during this rebuild procedure. It seemed too high given my experience. Hence I've looked up the reliability section of the datasheet for the new P300-series, 2 TB Toshiba desktop-class hard drive I just bought:

I'm a bit suspicious of the way probability is specified here. Strictly reading the exponential notation, 10E14 means that the probability of an unrecoverable error (URE) is one error per 10⋅10¹⁴ bits. Expressed as probability of an error when reading a single bit:

In another datasheet for a different series of drives (however this time for data center instead of consumer use) the error rate is given as 10 errors per 10¹⁶ bits. This again gives the same error probability of 10^-15.

Consider this probability for a second. It's such a fantastically low number. I don't remember ever encountering an actual technical specification that would involve a probability that has a one preceded by fifteen zeros - or in other words - fifteen nines of reliability.

The number is just on the edge of what you can represent with the common 64-bit double-precision floating-point format. If using a tool like numpy that only uses double-precision, any calculations with such values need to be done extra carefully to ensure that loss of numerical precision doesn't lead to nonsensical results.

Hard drives tend to use SI prefixes instead of binary, so I'll do the calculation for 4 terabytes instead of 4 tebibytes like it says in the quote:

For this calculation it doesn't matter whether we're reading this number of bites from one or two drives since the URE probabilities are assumed independent. The probability of getting at least one error during the rebuild is:

Note that if I read 10E14 in the original reliability specification as 10¹⁴, the probability of a rebuild error goes up to 27%.

This comes out a bit more optimistic than the higher-than-50% figure given in the warning, at least for this specific series of hard drives. I guess whether 3.1% is still too high depends on how much you value your data. However consider that in the original scenario this is the probability of an error given that another hard drive in the array has already catastrophically failed. So the actual probability of data loss is this multiplied with the (unknown) probability of a complete drive failure.

Then again, consider that this is a desktop drive. It is not meant to be put into a RAID array and is typically used without any redundancy. Some people will even scream at you if you use desktop drives in a RAID due to timeout issues. Without any redundancy this probability directly becomes the probability of data loss. And that seems exceedingly high - especially considering that drives up to 8 TB seem to be sold with this same error rate specification. Even with that amazing reliability of reading a single bit, modern drives are simply so large that the vanishingly tiny error probabilities add up.

Two weeks ago I was writing about 3D-printable replacements for 3.5" drive rails used in an old Chieftec PC enclosure I have. The original plastic rails became brittle with time. They often broke when I was replacing hard drives and I've eventually ran out of spares. I've drawn up a copy in FreeCAD and modified the design so that it was easily printable on a FDM printer. At the time of my last post I was waiting to get a sample pair printed. This is a quick update on that. I've tried the new rails and found that they fit well, but need some minor improvements.

This is how the new rails look when mounted on a 3.5" hard drive. Unfortunately I've specified a wrong diameter for the holes in the STL file, so the imperial-sized screws that go into these drives don't fit. I've manually drilled the holes with a larger diameter on the photo above, but obviously it's better if the rails come correct from the printer in the first place.

These pieces were kindly printed for me by Matjaž on a Prusa i3 MK3S in PETG filament. They feel sturdy and more than strong enough for their purpose. The only thing I was slightly worried about is them getting softer when mounted on warm disk drives. According to the Prusa website the material should be good enough up to 68°C. My monitoring says that no drive got hotter than 50°C in the last 12 months so even in summer they should be fine.

I was happy to see that the drive with the new rails fits into the case perfectly. Much better in fact than with the original rails. Original rails were very difficult to handle, even when new. These slide in and out with just enough force. The plastic arcs on the top and bottom of the rails engage nicely with the guides and provide enough friction so that the drive doesn't rattle in the case. The strengthened tabs on the side also fit nicely. I saw no need to fix any of the basic dimensions.

Apart from the holes the only other part that needed fixing is the flexible latch. This latch has to be printed in a separate piece. My original idea was that I will glue it to the base part of the rail. However the latches all broke off when I was testing them. Part of the problem was that the cyanoacrylate (superglue) I was using doesn't seem to have good adhesion to PETG plastic. I'll probably find a glue that works better, but I still wanted to change the design so that it depends less on the strength of the bond between the two plastic pieces.

The picture below shows how I've slightly modified the latch. The red part is the latch and purple is the base part of the rail. This picture shows a cross-section in FreeCAD. See my last post for the renders of the complete rail.

The new latch design (top) has a tab that inserts into a slot in the rail. When the latch bends, the slot itself should take most of the torque and hold it in place. The glue should only hold it so that the latch doesn't fall out of the slot when not under tension. In the old latch design (bottom) the glue itself was taking all of the torque when the latch was bending.

I would be even happier with a design that wouldn't require glue at all and where the latch would click into place somehow. However I felt that coming up with the right tolerances for that would require several more round trips between FreeCAD and the printer and, since I'm already at version 4 of the design, I didn't want to waste any more time on this.

Anyway, I've put the updated STL files at the same place as last time. Again I'm waiting for the printouts. Hopefully when I get the chance to test them no further changes will be necessary and I can finally install a new stack of hard drives into my case.

I recently had to replace an old Belkin UPS that had developed some weird problems. After some searching around for a suitable replacement I bought an APC Back-UPS 700VA (model BX700UI) from a local web shop. Even though it's not listed under the Network UPS Tools hardware compatibility list, getting it to run under Debian was relatively simple. Still, here are some random notes about it in case anyone else finds them useful.

Image by Schneider Electric

As this blog post reports, the UPS is fully supported by the usbhid-ups driver in NUT. I'm currently using version 2.7.4 as packaged in Debian Stretch. Apart from setting the driver in /etc/nut/ups.conf as described there is no other special NUT configuration necessary.

I also had to add an udev rule to change the owner and permissions of the USB device so that NUT could access the UPS. The correct rule can be found in this answer. Note that the "could not detach kernel driver from interface" part of the error message you get if this rule is missing is somewhat misleading.

I've been testing this UPS for around a week on a test setup and I didn't find any problems with the basic functionality. Switch over to battery works fine. The system shuts down cleanly when the UPS reports battery low and correctly turns off the UPS output afterwards. It also comes back on when the power returns.

My old Belkin UPS seemed to impose a hard 30 minute battery run time limit. It would not run the load for more than 30 minutes and would report that the battery is empty regardless of its state. This UPS doesn't have that limitation and will happily run on battery for hours if the load is low enough.

I did increase the battery.charge.low and battery.runtime.low to 20% and 5 minutes respectively. These settings determine when the system will consider the battery depleted and shut itself down. While I didn't find any problems with the defaults (10% and 2 minutes), they seemed to leave very little reserve. This reserve battery charge comes handy if the line power comes back for a minute and goes away again. In this case the UPS must have enough run time left to boot the system back up and shut it down again. Since the battery is large enough for my use case I thought increasing this safety margin made sense.

Note that setting the variables using upsrw requires that a upsd user is both defined in /etc/nut/upsd.users with the permission for the SET action and also has a permission to connect to upsd in /etc/hosts.allow. I overlooked that latter part and spent some time banging my head over the ERR ACCESS-DENIED error.

I'm not sure how persistent the settings done with upsrw are. They are supposed to write the values to the non-volatile memory in the UPS so they should only need setting once. They should survive across reboots and power losses, but so far I'm not totally convinced they will not revert back at some random time in the future.

As far as diagnostic information goes, this UPS isn't very generous. It doesn't measure line frequency, temperature or output voltage. It does measure input voltage and it returns reasonable values for it when the line power is present, but I noticed that it reports 256 V when the input voltage is in fact 0 V.

I had a quick look at the raw USB HID traffic from the UPS and I didn't find anything particularly interesting. At least it doesn't look like the relative lack of diagnostics is due to missing support in NUT.

The battery takes approximately 3 hours to fully recharge. I noticed some very faint crackling noise from it when it's charging. To me it hears like small bubbles popping when I press my ear against the case. This noise is distinct from the mains hum and doesn't hear like a DC-DC converter running. I don't have much experience with lead-acid batteries, so this is probably normal behavior. I just never noticed it before. Apparently even sealed gel batteries form some hydrogen gas that is then recombined inside with a catalyst.

I use an old Chieftec tower case for my home desktop computer. Instead of the usual screws the case uses purple plastic retainers for mounting 3.5" disk drives. These parts were quite fragile and even though the case came with plenty of spares I broke all of them over the years. I failed to find an off-the-shelf replacement so I went and designed a 3D printable part to replace them. I'm currently waiting to get the prototype printed.

I don't know the model name of the case or whether Chieftec had a specific name for this method of mounting drives. The only identifiable mark on the case is the name "Chieftec" embossed in the front panel. I wrote about it in this blog post so it must be at least 16 years old at this point. After quite a lot of web searching, this 2004 review of the BX-03BL-BL-BL model was the only other reference to this rail design I could find. It has a photo of the rails that look identical to the ones I have although the design of the rest of the case looks different.

I think these plastic rails were badly designed from the start. I remember even when the case was new it was very hard to insert or remove a drive. The rails are formed in the shape of a slight arc. When inserting a drive into the slot they straighten out which grips the drive in place. However the rails are fixed to the drive at two points so they can't flex freely. The other problem is that the triangular catch that latches into the metal case has no spring. You need to bend a relatively short and thick length of plastic to unlatch.

The plastic didn't age well. It got quite brittle and also changed color from a bright pastel purple to this purplish-gray. Very soon changing a hard drive in the case meant breaking one or both of the rails that held it in place. These days it begins to crack if I put even a slight pressure on it. An unfortunate flaw since otherwise I like this case and it has served me well for all those years.

The rails have a complex shape that isn't printable on a common FDM printer in one piece. To get a single, flat face that starts on the printer's build plate I removed the outward arc in the base shape. It was problematic anyway. Since holes for the sunken screw heads require an overhang in this orientation, I left a thin sacrificial layer that separates the two hole diameters. This needs to be drilled through before the screws can be inserted. I plan to use normal screws instead of the knurled pins the original rails used.

I moved the catch to a separate part that must be glued to the base part. It's mounted on a thin leaf spring that should provide the flexibility I was missing in the original. This part can be printed with layers following the spring shape which should make it more resistant to bending. I also extended the tabs on the sides to make them stronger. After the handle these were the second most common thing to break when unlatching the rail.

This was the first more complicated mechanical part I designed using the FreeCAD's Sketcher Workbench. I like the visual way of defining lengths and constraint on shapes mapped to object's faces. It's much clearer than the approach I took with my previous 3D printing projects - using cubes and other basic primitives from the Part Workbench, placing them using Python expressions and using unions and subtractions to get the final shape.

The problem I encountered however was that Sketcher isn't very convenient for experimenting. Because of the topological naming problem you can't easily go back and change things on a model. So far I've already ended up redrawing this shape 3 times from bottom up. Changing something in the sketch for the base shape tends to invalidate all other sketches for pockets and pads that are built on top of it. I don't know how to fix the missing edge and face references without starting all over again.

I'm now waiting to get these new rails 3D printed and I'm yet to see how they turn out in practice. This is the third iteration and I think I got all the measurements right. I'm mostly concerned if the new rails will fit snugly enough into the case to prevent the drive rattling and at the same time not be too hard to insert and remove. If they are too loose I may just add some rubber padding like the 2004 review I linked above suggests.

I've made the current version of my 3D printable design available for download here. I'll write another post when I have the chance to test these out and if I make any more iterations to the design.

What can I say about the last year that hasn't already been said a hundred times over? It's been a year of restrictions, lockdowns and changing habits. I'm immensely thankful that everyone close to me has managed to avoid the pandemic so far. It has not been easy for anyone. The quickly changing information on the virus was a constant source of doubts in what was a good balance between keeping safe and retaining some resemblance of normal life.

I don't feel I have much right to complain though. Compared to other personal stories I've heard I'm lucky. I don't have many responsibilities that would require a lot of personal contacts. I work for a company that has recognized the seriousness of the situation very early on and switched largely to remote work where possible.

It's been a year of sitting alone at my home desk. I've never been a particularly social person and I usually enjoy working alone, but the isolation this year sometimes felt painfully strong. I feel like the social distancing didn't just apply to physical contacts and we've all grown further apart even with all the video, voice and text communication within hand's reach. It was easy to get unreasonably angry over people holding different opinions on what risks they were prepared to take with their own health and the health of others.

Reduced work and the general chaos interrupting other plans meant that I worked a lot on personal projects. I've expanded the instrumentation in my home electronics lab: I've finished my home-brew curve tracer for non-linear device measurements. I didn't write much about that on the blog. I still have some unpublished notes on an interesting detail of bipolar junction transistors, which was the original motivation for making the instrument in the first place. Maybe I'll find the time to write it up next year.

On the other hand I did write a series of posts on RF vector measurements. I'm very happy with how my SDR-based system turned into a small, versatile instrument. With a NanoVNA available at perhaps a tenth of what I spent in total it might not look like the project made much economic sense. However I found that my instrument complements it nicely, with better characteristics in some places, worse in others and much more flexibility. It seems to fit nicely between the extremely low cost NanoVNA and the astronomical price of a professional instrument. Not to mention that I learned a lot while making it and refreshed my knowledge from the under-grad microwave engineering course I took long ago.

I've again drawn a lot, more than any other year despite the arts classes being interrupted. It was my almost daily activity in the evenings after I got fed up with electronics. In May, my comic Social distancing won the second place in its category in the Life in the time of coronavirus competition organized by the National Museum of Contemporary History. With the bird flu epidemic hitting local bird populations later in the year, my two quarreling crows might have been an even better allegory for the times than I originally intended.

In general I wanted to give some more meaning to the illustrations and characters I've drawn this year. This led me to write a few short stories. Looking back, these were probably more things I needed to do for myself to deal with various frustrations and anxieties. I've only published one so far here on this blog and even that was quite an internal struggle to do. I don't know about the others. I still like the pictures that go with them, but the text would need significant editing before I would be comfortable with putting them out in public.

The web was full of people cheering for 2021 to come as quickly as possible, as if viruses and other such things pay any attention to the numbers on the calendar. I have some ideas what I want to do next year: I want to keep drawing and, considering I'll be using my home desk as the company hardware lab for some time yet, I have some further ideas on new instruments I want to make. However I can't shake the feeling that the freedom to pursue my personal projects in relative idleness during lockdowns was on borrowed time and that I'll have that much less time for such things in the future. We'll see. If we all remain healthy it will be a good year.

The batteries in the remote control for my DVD player leaked. I thought it would be best to clean it out thoroughly as soon as possible. The electrolyte from the alkaline batteries attacks the phenolic paper substrate that is used for the cheap circuit boards. If left uncleaned the circuit board delaminates and basically falls apart after a few years. This is what happened to the remote for my old Hi-Fi and the circuit is more or less irreparable after that happens.

This is the remote model RC-5340 that came with the Philips DVP3360 DVD player.

Cleaning all the mess left by the batteries required opening the plastic enclosure. I couldn't do a good job through the battery compartment alone. Unfortunately the enclosure is really tightly held together with plastic tabs. I found it impossible to pry open without damage using my normal tools. There is almost no gap between the top plate and the lower part. I finally managed to open it with the iFixit Jimmy metal spudger tool kindly borrowed from Matjaž.

The best strategy was to find a weak spot along the longer sides and then work the gap from there. You can see on my photos where the plastic tabs are located. The tabs in the top and bottom sides are so tight that I couldn't open the enclosure without breaking some of them.

The chip is marked "ET3013 0907".

I scrubbed the whole thing with soap and water, dried it and also cleaned the rubber contacts with ethanol. There are already some dark spots on the PCB where the electrolyte has started to attack the board. I hope those won't spread too much.

Recently I noticed some unusual red dots and lines on the printouts from my Epson L310. It was interesting because it seemed like extra dots were getting added to the paper. This is just the opposite of the usual problem of clogged nozzles. In that case, dots (or whole lines) are missing on the paper. The extra dots also looked well-defined and didn't look like ink was just getting smudged onto the paper by a dirty print head.

I've ran the nozzle check using escputil -n and this is how the test area looks like:

It looks like one magenta nozzle is spewing ink at an angle. At first I thought it was an electrical problem and somehow one nozzle got shorted to another. But this looks more like the ink from that nozzle gets deposited onto the paper a bit to the top and left from where it should be. The line from that nozzle looks a bit more blurry than the other ones, which also suggests that something is interfering with the ink flow.

I don't remember seeing something like that before. Anyway, I just thought that was interesting. So far it didn't bother me too much and maybe it will go away during printer's regular self-cleaning cycles. If it doesn't fix itself I'll probably try to clean it up manually.

Back in 2013 I bought the CubieTruck, a single-board computer based on the Allwinner A20 ARM system-on-chip. Soon after I got it up and running with Debian I began using CubieTruck as a tiny home server. I eventually migrated this blog to it as well after my other server was knocked off-line by ice. It's been happily running on a shelf for almost 7 years now and has been an incredible value for its original 100 € price. It has been showing it's age though and with Debian Jessie running out of security support it was time to move on. I had a spare Zotac CI327 ready as a replacement and a few weeks ago I finally managed to move the last services off CubieTruck.

My CubieTruck really had a very good track record as far as reliability goes. It currently has a 464 day uptime and in recent memory only got reset because of a short power outage. In its early days I had problems with CRC errors on the SATA bus. This mostly got resolved after I changed to stock cable that came with it with a different one. At least it never again caused filesystem corruption (that I know of). The UDMA CRC error count reported by SMART currently sits at 276. It still incremented by 1 every once in a while, most often when I was doing something in the physical vicinity of the CubieTruck. I suspect the SATA signal integrity was still pretty marginal, even with a different cable.

Unfortunately, the long uptime is also a symptom of the largest problem I had with it: hugely outdated kernel. The SoC requires running a special Allwinner fork of the Linux kernel and it was soon severely outdated. I'm very grateful that Daniel Andersen kept the kernels updated with upstream patches for as long as the 3.4 branch was officially supported by kernel developers. Still, the last released one which is running on it right now, 3.4.112, is ancient by modern standards.

There was an effort to merge all the required code to the upstream kernel. As far as I remember it did come to a point where running the upstream kernel was perfectly viable, but I never did it. The problem was that NAND flash was never well supported. CubieTruck boots off 8 GB of on-board flash. Without support for it in the kernel you can't access the boot partition from Linux, which in practice means no easy kernel updates, which is the whole point. The alternative is to boot off an SD card, but I wanted to avoid that at all costs.

The machine I'm replacing CubieTruck is a Zotac Zbox nano CI327. It is itself outdated at this point and has a quad-core Celeron N3450 and 4 GB of RAM. I've also installed a Kingston 120 GB SSD (SA400S37120G). Unfortunately I haven't done a comprehensive HTTP benchmark comparison like I did last time I moved my blog installation. Mostly because I forgot about it, but there's also the fact that I don't know how to properly benchmark a HTTP service anymore. Apache Benchmark I used last time doesn't give representative results because it doesn't support SSL session caching.

I did measure how long it takes to rebuild the blog though. This is a single-threaded Perl process that builds around 25 MB of HTML files. The following shows user space time as reported by the time utility. I recorded the fastest run out of three on each machine:

As you can see, Zotac really leaves CubieTruck in the dust in this test. The CubieTruck numbers here aren't comparable to my old test. Even though the method was the same, my blog now has more entries and hence takes more effort to build. Also, in my previous test the CubieTruck was still running of an SD card, before I installed a SATA SSD.

As far as HTTP performance goes, I did notice this nice drop in the page load time graph after migration to the Zotac. This is from a Munin plug-in that runs on some other host on the Internet and requests my blog's homepage. On the other hand, server's 90 percentile response times for regular traffic haven't changed noticeably. In any case, CubieTruck was already fast enough to handle HackerNews home page traffic without major problems.

In conclusion, CubieTruck has been a joy and one of those machines that leave a lasting good impression. Even though it's inviting to scavenge its SSD I will very likely keep it around in case I need an ARM build machine or something similar. As far as I know it is still one of the rare ARM-based single-board computers that has decent storage performance due to it's native SATA interface. It's unfortunate that later Allwinner SoC versions abandoned it. It was only recently that the new Raspberry Pis with USB 3 got significantly better throughputs, although I would still like to see some long-term reliability figures for USB 3-attached storage.

How can you tell if someone uses a Bullet Journal®? You don't have to, they will immediately tell you themselves.

Some time last year I saw this book in the window of a local bookstore. I was aware of the website, but I didn't know the author also published a book about his method of organizing notebooks. I learned about the Bullet Journal back in 2014 and it motivated me to better organize my daily notes. About 3000 written pages later I'm still using some of the techniques I learned back then. I was curious if the book holds any new useful note-taking ideas, so I bought it on the spot.

The Bullet Journal Method is a 2018 book by Ryder Carroll (by the way, the colophon says my copy is printed in Slovenia). The text is split into 4 parts: first part gives motivation for writing a notebook. That is followed by a description of the actual note-taking methods. The third and longest part of the book at around 100 pages is called "The Practice". It's kind of a collection of essays giving advice on life philosophy with general topics such as meaning, gratitude and so on. The last part explores a few variations of the methods described in the book.

The methods described in the book differ a bit from what I remember. In fact the author does note in a few places that their advice has changed over time. The most surprising to me was the change from using blank squares as a symbol for an unfinished task to simple dots. The squares were in my opinion one of the most useful things I took from the Bullet Journal as they are a very clear visual cue. They really catch the eye among other notes and drawings when browsing for things left undone in a project.

In general, the contents of my notebooks are quite different from the journals the book talks about. I don't have such well defined formats of pages (they call it "collections"), except perhaps monthly indexes. My notebooks more resemble lab notes and I also tend to write things in longer form than the really short bullet lists suggested in the book. The author spends a lot of time on migrations and reflection: rewriting things from an old, full notebook to a new one, moving notes between months and so on. I am doing very little of that and rely more on referencing and looking up things in old notebooks. I do see some value in that though and after reading the book I'm starting to do more of it for some parts of my notes. I've experimented with a few other note-taking methods from the book as well, and some seem to be working for me and I've dropped the others.

I was surprised to see that a large portion of the book is dedicated to this very general motivational and life style advice, including diagrams like the one you see above, much in the style of self-help books. It made me give up on the book midway half-way through for a few months. I generally have a dislike for this kind of texts, but I don't think it's badly written. The section is intertwined with exercises that you can write down in your journal, like the "five whys" and so on. Some were interesting and others not so much. Reading about a suggestion to write your own obituary after a recent death in the family was off-putting, but I can hardly blame the book for that coincidence.

There is certainly some degree of Bullet Journal® brand building in this book. It feels like the author tries quite hard to sell their method in the first part of the book via thankful letters and stories from people that solved various tough life problems by following their advice. Again, something I think commonly found in self-help books and for me personally this usually has the opposite effect from what was probably intended. I do appreciate that the book doesn't really push the monetary side of it. Author's other businesses (branded notebooks and the mobile app) are each mentioned once towards the end of the book and not much more.

Another pleasant surprise was the tactful acknowledgment from the author that many journals shared on the web and social media don't resemble real things and can be very demotivational or misleading. I've noticed that myself. For example, if you search for "bullet journal" on YouTube you'll find plenty of people sharing their elaborately decorated notebooks that have been meticulously planned and sectioned for a year in advance. That's simply not how things work in my experience and most of all, I strongly believe that writing the notebook with the intention of sharing it on social media defeats the whole purpose.

In conclusion, it's an interesting book and so far I've kept it handy on my desk to occasionally look up some example page layouts that are given throughout it. I do recommend it if you're interested in using physical notebooks or are frustrated with the multitude of digital productivity apps that never tend to quite work out. It's certainly a good starting point, but keep in mind that what's recommended in there might not be what actually works best for you. My advice would be only to keep writing and give it some time until you figure out the useful parts.

One of my more vivid childhood memories is from the center of Ljubljana, somewhere around the new year 1990. On the building of the Nama department store there was a new, bright red LED screen. It was scrolling a message that read "Welcome to the new decade". It was probably one of the first such displays I ever saw. I remember finding the message kind of weird and surprising. I think up to that point I had decades shelved among the terms that were only the matter of books or movies.

It's now the end of another decade and, amusingly enough, I was again not really thinking about it in such terms. I was only reminded of the upcoming round number on the calendar when social media posts and articles summarizing the past 10 years started popping up. Anyway, I'm not even going to attempt to sum up the decade. A huge enough number of things happened in the past year alone, both happy and sad and somewhere in between. I usually have problems summarizing even that down to a few paragraphs, so here are only a few personal highlights.

On the electronic side, I'm really happy with how one work-related project turned out. It's a small multi-purpose microcontroller board that includes an analog front-end for certain proprietary buses. I've executed the whole project, from writing up a specification based on measurements and reverse engineering, drawing up the schematic to assembling prototypes and coding the firmware. It was an interesting exercise in optimization, both from the perspective of having a minimal BOM and low-level programming of integrated peripherals in the microcontroller.

Each year I mention the left-over pile of unfinished side projects. This year isn't any different and some projects stretch back worryingly deep into my stack of notebooks. Perhaps the closest to completion is a curve tracer that I designed while researching a curiosity in bipolar transistor behavior. I've also received the ERASynth Micro signal generator that I've helped crowdfund. It's supposed to become a part of an RF measurement system that I'm slowly piecing together. I feel bad for not posting a review of it, but as usual other things intervened.

I've continued to spend many evenings drawing, either in a classical drawing class at the National Gallery or behind the digital setup I have at home. At the start of the year I was doing some more experiments with animation, trying out lessons I've learned and checking out how far I can get with Python scripts for compositing and lighting effects. I further developed my GIMP plug-in.

I played around with some story ideas, but I didn't end up doing any kind of a longer project like I did a year I ago. I enjoyed trying out different drawing styles, experimenting with character design and doing random illustrations that came up in my mind. I've come up with some kind of an alternative space-race theme with animals, but in the end I realized that while I can draw the characters, I don't really have a story to tell about them.

Speaking about telling stories, I've written more blog posts this year than the year before. I've also had my moment of fame when my rant about Google blocking messages from my mail server was posted on HackerNews and reached the top of the front page. My server received a yearly amount of traffic in just a couple of days and the article got mentioned on sites like Bloomberg and BoingBoing. It was a fresh dose of motivation to keep writing amid the trend of falling number of visitors and RSS subscribers that I've seen since around 2014.

As I always repeat in these posts, it's hard to sum up 365 days in a few paragraphs. I've tried to stick to the positive side of things above. In case the picture is too rosy, I must add that there were also sad times that were hard to wade through and plans that didn't turn out as they should. The next year looks like it will bring some big challenges for me, so again I'll say that I won't make any plans on what kind of personal projects I'll do. If anything, I wish to finish some that I already started and try to refrain from beginning any new ones to add to the pile.

Some time ago I made a decision to start bringing my own lunch to work. The idea was that for a few days per week I would cook something simple at home the evening before and store it in a plastic container over night in my fridge. I would then bring the container with me to the office next day and heat it up in a microwave at lunch time. For various reasons I wasn't 100% consistent in following this plan, but for the past 3 months or so I did quite often make use of the oven in the office kitchenette. Back in mid-September I also bought a new set of food-grade plastic containers to use just for this purpose.

Around a week ago, just when I was about to fill one of the new containers, I noticed some white stains on its walls. Some increasingly vigilant scraping and rinsing later and the stains started looking less and less like dried-on food remains and more like some kind of corrosion of the plastic material. This had me worried, since the idea that I was eating dissolved polymer with my lunch didn't sound very inviting. On the other hand, I was curious. I've never seen plastic corroding in this way. In any case, I stopped using the containers and did some quick research.

After carefully inspecting all plastic containers in my kitchen, I've found a few more instances of this exact same effect. All were on one of the two types of containers I've used for carrying lunch to work. The two types are shown on the photo above. The top blue one is a 470 ml Lock & Lock (this model is apparently now called "classic"). It's dated 2008, made in China. I have a stock of these that I've used for more than 10 years for freezing or refrigerating food, but until recently never for heating things up in a microwave. The bottom green one is a 1.1 L Curver "Smart fresh". I've bought a few of these 3 months ago and only used them for carrying and heating up lunches in a microwave.

Both of these types are marked microwave safe, food safe and dishwasher safe (I've been washing the containers in a dishwasher after use). They all have the number "5" resin identification code and the PP acronym, meaning they are supposed to be made out of polypropylene polymer. The following line of logos is embossed on the bottom of the Curver containers (on a side note, the capacity spelled out in Braille seems to say "7.6 L"). Lock & Lock has a similar line of logos, except they don't advertise "BPA FREE":

The damage to the Curver container is visible on the photograph below. It looks like white dots and lines on the vertical wall of the container. At a first glance it could be mistaken for dried-on food remains. On all damaged containers it most often appears in a line approximately around the horizontal level where the interface between the liquid and air would be. I tend to fill these to around the half way mark and I've used the containers both for mostly solid food like rice or pasta and liquids like sauces and soups. If I run a finger across the stains they feel rough compared to the mirror finish plastic in the other parts. No amount of washing with water or a detergent will remove them. However, the stains are much less visible when wet.

Here is how these stains look under a microscope. The width of area pictured here is approximately 10 mm. Microscope shows much better than the naked eye that what look like white stains on the surface are actually small pits and patches of the wall that became corrugated. The damage does appear superficial and doesn't seem to penetrate the immediate surface layer of the plastic.

I've only used the containers in this office microwave. It's a De'Longhi Perfecto MW 311 rated at 800 W (MAFF heating category "D"). I've always used the rotating plate, usually the highest power level and 2 to 3 minutes of heating time per container.

After some searching around the web, I found a MetaFilter post from 2010 that seems to describe exactly the same phenomenon. Rough patches of plastic that seem like corrosion appearing on Lock & Lock polypropylene containers. The only difference seems to be that in Hakaisha's case the damage seems to be on the bottom of the container. The comments in that thread that seem plausible to me suggest physical damage from steam bubbles, chemical corrosion from tomato sauce or other acids in food or some non-specific effect of microwaves on the plastic.

My experience suggests that heating and/or use in a microwave oven was required for this effect. If only food contact would be to blame, I'm sure I would have seen this on my old set of Lock & Lock containers sooner. Polypropylene is quite a chemically inert material (hence why it's generally considered food safe), however its resistance to various chemicals does decrease with higher temperatures. For example, the chemical resistance table entry for oleic acid goes from no significant attack at 20°C to light attack at 60°C.

The comment about tomatoes is interesting. I've definitely seen that oily foods with a strong red color from tomatoes or red peppers will stain the polypropylene, even when stored in the refrigerator. In fact, leaflets that come with these food containers often warn that this is possible. In my experience, the red, transparent stain remains on the container for several cycles in the dishwasher, but does fade after some time. My Lock & Lock containers have been stained like that many times, but didn't develop the damaged surface before I started microwaving food in them.

Physical damage from steam bubbles seems unlikely to me. I guess something similar to cavitation might occur as a liquid-filled container moves through nodes and antinodes of the microwave oven's EM field, causing the water to boil and cool. However, it doesn't explain why this seems to mostly occur at the surface of the liquid. Direct damage from microwave radiation also doesn't make sense. It would occur all over the volume of the plastic, not only on the inner surface and in those specific spots. In any case, dielectric heating of the polypropylene itself should be negligible (it is, after all, used for low-loss capacitors exactly because of that property).

Another interesting source on this topic I found was a paper on deformation of packaging materials by Yoon et al. published in Korean Journal of Food Science and Technology in 2015. It discusses composite food pouches rather than monolithic polypropylene containers, however the inner layer of those pouches was in fact a polypropylene film. The authors investigated the causes of damage to that film after food in the pouches has been heated by microwaves. They show some microphotographs that look similar to what I've seen under my microscope.

Unfortunately, the paper is in Korean except for the abstract and figure captions. Google translate isn't very helpful. My understanding is that they conclude that hot spots can occur in salty, high-viscosity mixtures that contain little water. My guess is the mixture must be salty to reduce the penetration depth due to increased conductivity and high-viscosity to lessen the effect of evaporative cooling.

Most telling was the following graph that shows temperature measurements in various spots in a food pouch during microwave heating. Note how the highest temperatures are reached near the filling level (which I think means on the interface between the food in the pouch and air). Below the filling level, the temperature never raises above the boiling point of water. Wikipedia has values between 130°C and 166°C for the melting point of polypropylene. Given the graph below it seems plausible that a partially-dried out food mixture stuck to the container above the liquid level might heat up enough to melt a spot on the container.

Image by Yoon et al.

In summary, I think spot melting of the plastic described in the Yoon paper seems the most plausible explanation for what I was seeing. Then again, I'm judging this based on my high-school knowledge of chemistry, so there are probably aspects of this question I didn't consider. It's also hard to find anything health- or food-related on the web that appears trustworthy. It would be interesting to try out some experiments to test some of these theories. Whatever the true cause of the damage might be, I thought it was prudent to buy some borosilicate glass containers to replace the polypropylene ones for the time being.

Yesterday I was invited to give a presentation at a seminar on procedures for product certification. My former colleagues at the Department of Communication Systems at the Jožef Stefan Institute invited three companies to share their experiences in getting electronics products to European market. We discussed compliance with the CE mark, testing for safety and electromagnetic compatibility standards and various other approvals that are required before you can put mass-produced electronics on the shelves.

In my presentation (slides are here) I've discussed my view of the two certification cycles I've participated in at Klevio during roughly the last year and a half. I did a short intro about the company and products and then listed what we chose to certify and how. I didn't do any general intro into the certification procedures and individual measurements. This in the end turned out just fine, because others did it better than I could. Most of the time I spent talking about purely practical lessons we learned on how to best prepare for the task. Looking back at it now, I feel like most of my advice boils down to having a good understanding of what is involved and not basing your expectations purely on certification lab's sales pitches.

I wasn't sure how much time I would have and what the interest of the audience was. Because of that I've put the details on debugging specific compliance problems into a separate part of the presentation. I've ended up doing that part of the talk as well. I discussed a problem we had with an ESD test temporarily disabling an audio codec IC and a problem with radiated emissions that took 3 months to debug and led to some pretty significant changes to one of the DC-DC converters.

I wanted to talk more about some interesting home-brew methods of estimating radiated emissions. I spent a lot of time researching and experimenting with them when I was debugging Klevio's compliance problems. In the end I realized that I could spend a whole talk just on that topic. It also turned out that none of those methods were actually useful in finding a solution, so it didn't make much sense to do more than just list them out. For more info on that, here's an article I found useful on making magnetic loop probes. The idea for the common-mode current probe based on a ferrite ring is from the Application Note AN045 by Richtek. The SDR-based method was my own idea based on my previous research on spectrum sensing and I might eventually do a longer write up on that in the future.

In the end, it was interesting to compare notes and what others have learned solving similar problems. I found that even though our EMI problem took longer to solve than others presented at the seminar, Klevio's experience didn't differ much in regard to timelines or certification lab practicalities. It seems almost everyone stumbles upon some problems during certifications, and while specific issues are unique, the biggest obstacle to finding a solution seems pretty universal: reproducing the problem outside of the certification lab.

Back in January I bought a new micro Hi-Fi after the transformer in my old one failed and I couldn't repair it. Since then I've been pretty happy using Denon RCD-M41DAB for listening to music and radio. I previously did some measurements on its output amplifier and found it meets its ratings in regards to power and distortion. In any case, that was more to satisfy my electrical engineering curiosity and I'm not particularly sensitive to reproduction quality. However after extended use two problems did pop up. Since I see this model is still being sold, I thought I might do a short follow up.

The first thing is that when streaming audio to the Hi-Fi over Bluetooth there are about 2 seconds of lag. It looks as if there's some kind of an audio processing buffer inside the RCD-M41DAB, or it might simply be the choice of the codec used for wireless transfer. When listening to music the delay isn't that annoying. For example, when clicking "next track" or changing volume on the music player on the phone it takes about 2 seconds before you can actually hear the change. However this lag does make a Bluetooth-connected RCD-M41DAB completely useless as an audio output when playing videos or anything interactive. The lag happens on iOS and Android and I'm pretty sure this is not something related to my smartphone, since when I'm using Bluetooth-connected headphones, there is no discernible lag there.

The other annoyance is that the CD player has problems reading some CDs in my collection. Apparently it reads the table of contents fine, because the number of tracks is displayed correctly. However it has problems seeking to tracks. For example, one disc won't play any tracks at all and I can just hear the head continuously seeking. Another disc won't start playing the first track, but if I skip it manually, player will find and play the second one just fine. All such discs play normally in the CD drive in my desktop computer and other CD players I've tried. It still might be that the discs have some kind of an error in them or have bad reflectivity (all of the problematic ones are kind of niche releases I bought from Bandcamp), but other players apparently are able to work around it.