Some SPF statistics

21.02.2016 19:04

Some people tend their backyard gardens. I host my own mail server. Recently, there has been a push towards more stringent mail server authentication to fight spam and abuse. One of the simple ways of controlling which server is allowed to send mail for a domain is the Sender Policy Framework. Zakir Durumeric explained it nicely in his Neither Snow Nor Rain Nor MITM talk at the 32C3.

The effort for more authentication seems to be headed by Google. That is not surprising. Google Mail is e-mail for most people nowadays. If anyone can push for changes in the infrastructure, it's them. A while ago Google published some statistics regarding the adoption of different standards for their inbound mail. Just recently, they also added visible warnings for their users if mail they received has not been sent from an authenticated server. Just how much an average user can do about that (except perhaps pressure their correspondents to start using Google Mail) seems questionable though.

Anyway, I implemented a SPF check for inbound mail on my server some time ago. I never explicitly rejected mail based on it however. My MTA just adds a header to incoming messages. I was guessing the added header may be picked out by the Bayesian spam filter, if it became significant at any point. After reading about Google's efforts I was wondering what the situation regarding SPF checks looks like for me. Obviously, I see a very different sample of the world's e-mail traffic as Google's servers.

For this experiment I took a 3 month sample of inbound e-mail that was received by my server between November 2015 and January 2016. The mail was classified by Bogofilter into spam and non-spam mail, mostly based on textual content. SPF records were evaluated by spf-tools-perl upon reception. Explanation of results (what softfail, permerror, etc. means) is here.

SPF evaluation results for non-spam messages.

SPF evaluation results for spam messages.

As you can see, the situation in this little corner of the Internet is much less optimistic than the 95.3% SPF adoption rate that Google sees. More than half of mail I see doesn't have a SPF record. A successful SPF record validation also doesn't look like that much of a strong signal for spam filtering either, with 22% of spam mail successfully passing the check.

It's nice that I saw no hard SPF failures for non-spam mail. I checked my inbox for mail that had softfails and permerrors. Some of it was borderline-spammy and some of it was legitimate and appeared to be due to the sender having a misconfigured SPF record.

Another interesting point I noticed is that some sneaky spam mail comes with their own headers claiming SPF evaluation. This might be a problem if the MTA just adds another Received-SPF header at the bottom and doesn't remove the existing one. If you then have a simple filter on Received-SPF: pass somewhere later in the pipeline it's likely the filter will hit the spammer's header first instead of the header your MTA added.

Posted by Tomaž | Categories: Life | Comments »

IEEE 802.11 channel survey statistics

14.02.2016 20:17

I attended the All our shared spectrum are belong to us talk by Paul Fuxjaeger this December at the 32C3. He talked about a kernel patch that adds radio channel utilization data to the beacons sent out by a wireless LAN access point. The patched kernel can be used with OpenWRT to inform other devices in the network of the state of the radio channel, as seen from the standpoint of the access point. One of the uses for this is reducing the hidden node problem and as I understand they used it to improve robustness of a Wi-Fi mesh network.

His patch got my attention because I did not know that typical wireless LAN radios kept such low-level radio channel statistics. I previously did some surveys of the occupancy of the 2.4 GHz band for a seminar at the Institute. My measurements using a TI CC2500 transceiver on a VESNA sensor node showed that the physical radio channel very rarely reached any kind of significant occupancy, even in the middle of very busy Wi-Fi networks. While I did not pursue it further, I remained suspicious of that result. Paul's talk gave me an idea that it would be interesting to compare such measurements with statistics available from the Wi-Fi adapter itself.

I did some digging and found the definition of struct survey_info in include/net/cfg80211.h in Linux kernel:

 * struct survey_info - channel survey response
 * @channel: the channel this survey record reports, mandatory
 * @filled: bitflag of flags from &enum survey_info_flags
 * @noise: channel noise in dBm. This and all following fields are
 *	optional
 * @channel_time: amount of time in ms the radio spent on the channel
 * @channel_time_busy: amount of time the primary channel was sensed busy
 * @channel_time_ext_busy: amount of time the extension channel was sensed busy
 * @channel_time_rx: amount of time the radio spent receiving data
 * @channel_time_tx: amount of time the radio spent transmitting data
 * Used by dump_survey() to report back per-channel survey information.
 * This structure can later be expanded with things like
 * channel duty cycle etc.
struct survey_info {
	struct ieee80211_channel *channel;
	u64 channel_time;
	u64 channel_time_busy;
	u64 channel_time_ext_busy;
	u64 channel_time_rx;
	u64 channel_time_tx;
	u32 filled;
	s8 noise;

You can get these values for a wireless adapter on a running Linux system using ethtool or iw without patching the kernel:

$ ethtool -S wlan0
NIC statistics:
     noise: 161
     ch_time: 19404258681
     ch_time_busy: 3082386526
     ch_time_ext_busy: 18446744073709551615
     ch_time_rx: 2510607684
     ch_time_tx: 371239068
$ iw dev wlan0 survey dump
Survey data from wlan0
	frequency:			2422 MHz [in use]
	noise:				-95 dBm
	channel active time:		19404338429 ms
	channel busy time:		3082398273 ms
	channel receive time:		2510616517 ms
	channel transmit time:		371240518 ms

It's worth noting that ethtool will print out values even if the hardware does not set them. For example the ch_time_ext_busy above seems invalid (it's 0xffffffffffffffff in hex). ethtool also incorrectly interprets the noise value (161 is -95 interpreted as an unsigned 8-bit value).

What fields contain valid values depends on the wireless adapter. In fact, grepping for channel_time_busy through the kernel tree lists only a handful of drivers that touch it. For example, the iwlwifi-supported Intel Centrino Advanced-N 6205 in my laptop does not fill any of these fields, while the Atheros AR9550 adapter using the ath9k driver in my wireless router does.

Short of the comment in the code listed above, I haven't yet found any more detailed documentation regarding the meaning of these fields. I checked the IEEE 802.11 standard, but as far as I can see, it only mentions that radios can keep the statistic on the percentage of the time the channel is busy, but doesn't go into details.

Weekly Wi-Fi channel statistic

Weekly Wi-Fi channel noise statistic

I wrote a small Munin plugin that keeps track of these values. For time counters the graph above shows the derivative (effectively the percentage of time the radio spent in a particular state). With some experiments I was able to find out the following:

  • noise: I'm guessing this is the threshold value used by the energy detection carrier sense mechanism in the CSMA/CA protocol. It's around -95 dBm for my router, which sounds like a valid value. It's not static but changes up and down by a decibel or two. It would be interesting to see how the radio dynamically determines the noise level.
  • channel_time: If you don't change channels, this goes up by 1000 ms each 1000 ms of wall-clock time.
  • channel_time_busy: Probably the amount of time the energy detector showed input power over the noise threshold, plus the time the radio was in transmit mode. This seems to be about the same as the sum of RX and TX times, unless there is a lot of external interference (like another busy Wi-Fi network on the same channel).
  • channel_time_ext_busy: This might be a counter for the secondary radio channel used in channel binding, like in 802.11ac. I haven't tested this since channel binding isn't working on my router.
  • channel_time_rx: On my router this increments at around 10% of channel_time rate, even if the network is completely idle, so I'm guessing it triggers at some very low-level, before packet CRC checks and things like that. As expected, it goes towards 100% of channel_time rate if you send a lot of data towards the interface.
  • channel_time_tx: At idle it's around 2% on my router, which seems consistent with beacon broadcasts. It goes towards 100% of channel_time rate if you send a lot of data from the interface.

In general, the following relation seems to hold:

\frac{t_{channel-tx} + t_{channel-rx}}{t_{channel}} < \frac{t_{channel-busy}}{t_{channel}} < 100\%
Posted by Tomaž | Categories: Life | Comments »