Avian’s Blog

Back in November I wrote about reliability problems with a bed-of-nails test fixture I've made for an electronic circuit. The fixture with 21 pogo pins only had around 60% long-term probability that all pins would contact their test pads correctly, leading to a very high false alarm rate. I did a quick review of blog posts about similar setups and scientific literature I've found on the subject. Based on what I've read it seemed that such severe problems were rare. From my own analysis I concluded that likely causes were either dirty test pads or bad contacts inside the pogo pins themselves, between the plunger and the body of the pin. The pogo pins I was using were on the cheaper end of the spectrum, so the latter explanation seemed likely.

Recently I got hold of a set of more expensive pins and, as it happened, also a new digital microscope. I was wondering how the mechanical design of the new pins compared to the old ones, so I looked at them under the microscope. This lead to some new clues about the cause of the problems I was investigating:

Pogo pin tips pictured above from left to right:

a) Harwin P19-0121, new (23.00 € for 10 pieces). Tip material is gold-plated steel.

b) P75-B1 type of uncertain origin, new (4.46 € for 10 pieces).

c) and d) two examples of P75-B1 removed from the test fixture after approximately 1500 mating cycles.

The more expensive Harwin pins show a significantly sharper point than the ones sold by Adafruit. Even when new, the cheaper pins have a slightly rounded tip. Over many mating cycles with a test pad the tips end up being even more flattened. The c) and d) pins above have been used with a flat test point surface on a lead-free HASL-finish PCB (the test setup described in my previous post). I couldn't find any specification of the longevity of the P75-series of pins. Harwin P19 are specified for 100k cycles, so it seems surprising that P75 would wear down so much after less than 2% of that amount. This evaluation by OKI shows that contact resistance of probes for wafer testing starts to rise somewhere after 10k cycles.

These flattened tips do explain somewhat the problem I'm seeing. Compared to sharp ones, dull or rounded contacts have a worse chance of piercing surface contamination on a PCB, like oxide or flux residue. Hence why my analysis showed that the failure rate was related to each production batch. Each batch had a slightly different amount of residue left on the boards and none was perfectly clean. First results show that replacing the pins did have a positive effect on the test reliability (I imagine it's hard to get any worse than that 40% fail rate), but I'll have to wait to get some statistically significant numbers.

In the context of using more expensive pogo pins, another issue came up. Some of the more expensive pogo pins use heads made from beryllium-copper alloy. None of the pins pictured above do, but other head shapes from the same Harwin P19 product line do in fact use beryllium according to their datasheets. Beryllium has some health risks associated with it, especially when it's in particulate form. I was wondering, if I switch the test setup to such pins, how much beryllium would be released into the environment due to parts wearing down in the way I've seen?

Image by Beryllium Science & Technology Association

From the microscope photographs above, I'm estimating that approximately 140000 μm³ of material was lost from one pin after 1500 cycles. This value is based on the volume of a cone that's missing above the flattened tips of pins c) and d) and probably overestimates the true amount. Given a BeCu alloy density of 8.25 g/cm³ and assuming beryllium content of 3% by mass, this comes out as approximately 0.04 μg of pure beryllium released into the environment. One figure I found for the recommended beryllium exposure per inhalable volume of air is 0.6 μg/m³.

This means that all accumulated dust from the wear of 15 pins would need to be distributed in a single cubic meter of air to reach the maximum recommended density for breathable air. Considering that the amount of wear shown above happened during a time span of months it seems unlikely that all of it would instantaneously end up gathered in a small volume. I don't know if the missing material would end up in the form of dust around the pins, or would be slowly carried away, smeared little by little on test pads. In any case, based on this back-of-the-envelope calculation beryllium contacts seem reasonably safe to use, even if the amount of beryllium lost isn't completely negligible compared to published exposure limits (but of course, I'm not any kind of a workplace safety expert).

I don't think this result is surprising. Finished products using beryllium are generally considered safe. BeCu alloys have been used for mundane things like golf clubs and musical instruments. Harwin doesn't publish any MSDS documents for their products. Also as far as I'm aware, beryllium use isn't covered by RoHS, REACH and other such regulations. But in any case, it can't hurt following some basic precautions when working with electronic components that incorporate this kind of materials.

Some time ago I made a decision to start bringing my own lunch to work. The idea was that for a few days per week I would cook something simple at home the evening before and store it in a plastic container over night in my fridge. I would then bring the container with me to the office next day and heat it up in a microwave at lunch time. For various reasons I wasn't 100% consistent in following this plan, but for the past 3 months or so I did quite often make use of the oven in the office kitchenette. Back in mid-September I also bought a new set of food-grade plastic containers to use just for this purpose.

Around a week ago, just when I was about to fill one of the new containers, I noticed some white stains on its walls. Some increasingly vigilant scraping and rinsing later and the stains started looking less and less like dried-on food remains and more like some kind of corrosion of the plastic material. This had me worried, since the idea that I was eating dissolved polymer with my lunch didn't sound very inviting. On the other hand, I was curious. I've never seen plastic corroding in this way. In any case, I stopped using the containers and did some quick research.

After carefully inspecting all plastic containers in my kitchen, I've found a few more instances of this exact same effect. All were on one of the two types of containers I've used for carrying lunch to work. The two types are shown on the photo above. The top blue one is a 470 ml Lock & Lock (this model is apparently now called "classic"). It's dated 2008, made in China. I have a stock of these that I've used for more than 10 years for freezing or refrigerating food, but until recently never for heating things up in a microwave. The bottom green one is a 1.1 L Curver "Smart fresh". I've bought a few of these 3 months ago and only used them for carrying and heating up lunches in a microwave.

Both of these types are marked microwave safe, food safe and dishwasher safe (I've been washing the containers in a dishwasher after use). They all have the number "5" resin identification code and the PP acronym, meaning they are supposed to be made out of polypropylene polymer. The following line of logos is embossed on the bottom of the Curver containers (on a side note, the capacity spelled out in Braille seems to say "7.6 L"). Lock & Lock has a similar line of logos, except they don't advertise "BPA FREE":

The damage to the Curver container is visible on the photograph below. It looks like white dots and lines on the vertical wall of the container. At a first glance it could be mistaken for dried-on food remains. On all damaged containers it most often appears in a line approximately around the horizontal level where the interface between the liquid and air would be. I tend to fill these to around the half way mark and I've used the containers both for mostly solid food like rice or pasta and liquids like sauces and soups. If I run a finger across the stains they feel rough compared to the mirror finish plastic in the other parts. No amount of washing with water or a detergent will remove them. However, the stains are much less visible when wet.

Here is how these stains look under a microscope. The width of area pictured here is approximately 10 mm. Microscope shows much better than the naked eye that what look like white stains on the surface are actually small pits and patches of the wall that became corrugated. The damage does appear superficial and doesn't seem to penetrate the immediate surface layer of the plastic.

I've only used the containers in this office microwave. It's a De'Longhi Perfecto MW 311 rated at 800 W (MAFF heating category "D"). I've always used the rotating plate, usually the highest power level and 2 to 3 minutes of heating time per container.

After some searching around the web, I found a MetaFilter post from 2010 that seems to describe exactly the same phenomenon. Rough patches of plastic that seem like corrosion appearing on Lock & Lock polypropylene containers. The only difference seems to be that in Hakaisha's case the damage seems to be on the bottom of the container. The comments in that thread that seem plausible to me suggest physical damage from steam bubbles, chemical corrosion from tomato sauce or other acids in food or some non-specific effect of microwaves on the plastic.

My experience suggests that heating and/or use in a microwave oven was required for this effect. If only food contact would be to blame, I'm sure I would have seen this on my old set of Lock & Lock containers sooner. Polypropylene is quite a chemically inert material (hence why it's generally considered food safe), however its resistance to various chemicals does decrease with higher temperatures. For example, the chemical resistance table entry for oleic acid goes from no significant attack at 20°C to light attack at 60°C.

The comment about tomatoes is interesting. I've definitely seen that oily foods with a strong red color from tomatoes or red peppers will stain the polypropylene, even when stored in the refrigerator. In fact, leaflets that come with these food containers often warn that this is possible. In my experience, the red, transparent stain remains on the container for several cycles in the dishwasher, but does fade after some time. My Lock & Lock containers have been stained like that many times, but didn't develop the damaged surface before I started microwaving food in them.

Physical damage from steam bubbles seems unlikely to me. I guess something similar to cavitation might occur as a liquid-filled container moves through nodes and antinodes of the microwave oven's EM field, causing the water to boil and cool. However, it doesn't explain why this seems to mostly occur at the surface of the liquid. Direct damage from microwave radiation also doesn't make sense. It would occur all over the volume of the plastic, not only on the inner surface and in those specific spots. In any case, dielectric heating of the polypropylene itself should be negligible (it is, after all, used for low-loss capacitors exactly because of that property).

Another interesting source on this topic I found was a paper on deformation of packaging materials by Yoon et al. published in Korean Journal of Food Science and Technology in 2015. It discusses composite food pouches rather than monolithic polypropylene containers, however the inner layer of those pouches was in fact a polypropylene film. The authors investigated the causes of damage to that film after food in the pouches has been heated by microwaves. They show some microphotographs that look similar to what I've seen under my microscope.

Unfortunately, the paper is in Korean except for the abstract and figure captions. Google translate isn't very helpful. My understanding is that they conclude that hot spots can occur in salty, high-viscosity mixtures that contain little water. My guess is the mixture must be salty to reduce the penetration depth due to increased conductivity and high-viscosity to lessen the effect of evaporative cooling.

Most telling was the following graph that shows temperature measurements in various spots in a food pouch during microwave heating. Note how the highest temperatures are reached near the filling level (which I think means on the interface between the food in the pouch and air). Below the filling level, the temperature never raises above the boiling point of water. Wikipedia has values between 130°C and 166°C for the melting point of polypropylene. Given the graph below it seems plausible that a partially-dried out food mixture stuck to the container above the liquid level might heat up enough to melt a spot on the container.

Image by Yoon et al.

In summary, I think spot melting of the plastic described in the Yoon paper seems the most plausible explanation for what I was seeing. Then again, I'm judging this based on my high-school knowledge of chemistry, so there are probably aspects of this question I didn't consider. It's also hard to find anything health- or food-related on the web that appears trustworthy. It would be interesting to try out some experiments to test some of these theories. Whatever the true cause of the damage might be, I thought it was prudent to buy some borosilicate glass containers to replace the polypropylene ones for the time being.

I have just released version 1.1.1 of the publicsuffix Python package. Baring any major bugs that would affect some popular software package using it, this will be the last release. I've released v1.1.1 because I received a report that a bug in publicsuffix package is preventing installation of GNU Mailman.

In the grand scheme of things, it's not a big deal. It's a small library with a modest number of users. I haven't done any work, short of answering mail about it, since 2015. Drop-in alternatives exist. People that care strongly about the issues I cover below have most likely already switched to one of the forks and rewrites that popped up over the years. For those that don't care, nothing will change. The code still works and the library is still normally installable from PyPi. Debian package continues to exist. The purpose of this post is more to give some closure and to sum up a few mail threads that started back in 2015 and never reached a conclusion.

I've first released the publicsuffix library back in 2011, two employers and a life ago. Back then there was no easily accessible Python implementation of Mozilla's Public Suffix List. Since I needed one for my work, I've picked up a source file from an abandoned open source project on Google Code (which was just being abandoned by Google around that time). I did some minor work on it to make it usable as a standalone library and published it on PyPi.

I've not used publicsuffix myself for years. Looking back, most of my open source projects that I still maintain seem to be like that. Even though I don't use them, I feel some obligation to do basic maintenance on them and answer support mail. If not for other reasons, then out of a sense that I should give back to the body of free software that I depend so much on in my professional career. Some technical problems are also simply fun to work on and most of the time there's not much pressure.

However one thing that was a source of long discussions about publicsuffix is the way the PSL data is distributed. I've written previously about the issue. In summary, you either distribute stale data with the code or fetch an up-to-date copy via the network, which is a privacy problem. These two are the only options possible and going with one or the other or both was always going to be a problem for someone. I hate software that phones home (well, phones Mozilla in this case) as much as anyone, but it's a problem that me as a mere maintainer of a Python library had no hope of solving, even if I got CC'd in all the threads discussing it.

The Public Suffix List is a funny thing. Ideally, software either should not care about the semantic meaning of domain names or this meaning should be embedded in the basic infrastructure of the Internet (e.g. DNS or something). But alas we don't live in either of those worlds and hence we have a magic text file that lives on a HTTP server somewhere and some software needs to have access to it if it wants to do its thing. No amount of worrying on my part was going to change that.

The other issue that sparked at least one fork of publicsuffix was the fact that I refused to publish the source on GitHub. Even tough there are usually several copies of the publicsuffix code on the GitHub at any time, none of them are mine. I was instead hosting my own git repo and was accepting bug reports and other comments only over email.

Some time ago already GitHub became synonymous with open source. People simply expect a PyPi package to have a GitHub (or GitLab, or BitBucket) point-and-click interface somewhere on the web. The practical problem I have with that is that it hugely increases the amount of effort I have to spend on a project (subjectively speaking - keep in mind this is something I do in my free time). Yes, it makes it trivial for someone to contribute a patch. However in practice I find that it does not result in greater quantity of meaningful patches or bug reports. What it does do is create more work for me dealing with low-effort contributions I must reject.

I'm talking about a daunting asymmetry in communication. Writing two sentences in a hurry in a GitHub issue or pushing a bunch of untested code my way in a pull request can take all of a minute for the submitter. On the other hand, I don't want to discourage people from contributing to free software and I know how frustrating it can be to contribute to open source projects (see my post about drive by contributions). So I try to take some time to study the pull request and write an intelligible and useful answer. However this is simply not sustainable. Looking back I also seem to often fail at not letting my frustration show through in my answer. Hence I feel like requiring contributors to at least know how to use git format-patch and write an email forms a useful barrier to entry. It prevents frustration at both ends and I believe that for a well thought-out contribution, the overhead of opening a mail client should be negligible.

Of course, if the project is not officially present on GitHub you get the current situation, where multiple public copies of the project still exist on GitHub, made by random people for their own use. These copies often keep in my contact details and don't obviously state that the code has been modified and/or is not related to the PyPi releases. This causes confusion, since code on GitHub is not the same as the one on PyPi. People also sometimes reuse version numbers for their own private use that conflict with version numbers on PyPi and so on and so on. It is kind of a damned if you do and damned if you don't situation really.

How can I sum this up? I've maintained this software for around 8 years, well after I left the company for which it was originally developed. During that time people have forked and rewrote it for various, largely non-technical reasons. That's fine. It's how free software is supposed to work and my own package was based on another one that got abandoned. I might still be happy to work on technical issues, but the part that turned out much more exhausting than working on the code was dealing with the social and ideological issues people had with it. It's probably my failing that I've spent so much thought on those. In the end, my own interests have changed as well during that time and finally letting it go does also feel like a stone off my shoulders.