Avian’s Blog

Thank you Borland, for making your antique Turbo C 2.01 compiler available on the internet as a free (as in beer) download.

I was stuck with my keylogger project. For some reason as soon as the size of the code for the 89C4051 microcontroller grew above 2048 bytes it stopped working. The program ran fine in the simulator but when I burned it to the actual chip, it remained completely dead.

I first suspected the old Keil compiler I'm using. But why would the code then run fine in the simulator? The programmer was also unlikely to cause this kind of a problem because the flash memory on the controller is programmed via a serial bus. So it is impossible for example for a single address line to loose connection.

At the end it turned out that the programmer software, a small DOS program, was only capable of dealing with 2 kilobytes of data at once. It took me so long to figure that out because the verify function also only compared the first 2 kb of the file on the disk with the first 2 kb of the flash memory. It looks like someone at Atmel really did not want to waste to much RAM.

After some searching on the Atmel website I found the C source code for this software, but without any instructions on which C compiler should I use. This string inside the compiled program gave me the first hint where to look:

MS Run-Time Library - Copyright (c) 1988, Microsoft Corp

So obviously this was compiled with a Microsoft C compiler. Now Microsoft C compilers for DOS vintage 1988 aren't exactly easy to find so I installed the best alternative I could find on my FreeDOS machine - Borland Turbo C. Of course the compiler then gave me a ton of errors. For example, Microsoft C obviously known C++ style comments back then, but Borland didn't. Also Microsoft C used some ugly-named functions for dealing with character graphics. Compare for example this from MS:

_clearscreen( _GCLEARSCREEN );

With the equivalent from Borland:

clrscr();

Luckily some search & replaces with the DOS version of VIM fixed that. It turned out however that a part of the program was written in assembler in a syntax that Borland C didn't recognize.

Some tweaking later I managed to successfully link the object file from Turbo C with the object file of the assembler part I got from Atmel that was made with Microsoft Assembler. I still can't believe that this worked. From my experience linking two object files even from different versions of the same compiler is asking for trouble.

After I got this working the solution to my problem was simply a matter of increasing a single constant from 2048 to 4096. So in case anybody else has the same problems, I'm putting the fixed compiled software on the net.

Reflections on Trusting Trust by Ken Thompson, one of the original authors of Unix, shows in a very nice way how you can't trust any binary if you can't trust the compiler that was used to compile it.

I think the part about the concept of a "learning" program is worth reading alone. I knew compilers are usually compiled by themselves (take GCC for example - it seems that the basic sign that a compiler is actually usable is that it can compile itself), but I never looked at this practice from the view point presented in the article.

I was planning to add a search function to tablix.org for some time. Specially the mailing list archive got so large that it is very time consuming to find posts about a specific subject.

All free search engines I found on the net resembled Google and other general internet search engines, which is not what I want. These search engines use a web crawler to index data on the web page which isn't very efficient if you want to index a trusted and a well known web site. Also simple web crawlers are bad because 1) They index all text (including menus, etc.) on web pages, not only content 2) They don't have any idea what exactly are they indexing. If you get a search hit, you have to check it manually if it is a news article, a mailing list post or a page in the documentation.

On the other hand, desktop searches, like Beagle are much more advanced - they use plugins to recognize and properly index each type of file on the filesystem. So I tried to make a web search engine that would be as flexible as that. It must use plugins to properly index mailing list archives, wiki pages, Nanoblogger posts, articles, Doxygen reference, etc. I used this Beagle mockup from Beagle UI Hackfest as a guide.

The result: Nanobeagle.

It's written in Perl, uses Swish-e indexing engine and it's not yet very stable. It currently has three indexing plugins: Hypermail archives, Nanoblogger news and Nanoblogger static articles. Icons are from the gnome-icon-theme part of the GNOME CVS repository.

I've been getting some strange spam lately. While Thunderbird doesn't seem to have problems, it crashes my Mac OS X Mail. Each time I open the Junk mail box, the application crashes with Sorting messages displayed on the status line. It doesn't respond to the Quit command and I have to use the Force quit from the Apple menu.

The solution I found at this time is to ctrl-click on Junk mailbox and select Delete junk mail. Mail will crash anyway, but next time I open it, the Junk mailbox is empty and the nasty mails are gone.

I'll see if I can find the specific message that causes this.

Do you want your code to be fast? Take my advice and don't even try hand optimizing it for these fancy SIMD instruction sets like MMX, SSE and similar. In fact, don't even google for "mmx optimization" or you'll be deceived by pretty numbers and benchmarks and you'll forget what you've read here. You will spend hours writing optimized memcpy() functions, parallelizing loops in your code and putting those damned PREFETCHes everywhere. And in the end (if you're anything like me), you'll rm -rf everything in frustration because no matter how you look at things, your cool assembly MMX optimized code will always run slower than plain dumb unoptimized C.

Or perhaps I'm doing something wrong here? Perhaps not. (No, I'm not mixing floating point and MMX!) Even AMD's optimization guide says that in most cases you should use scalar instructions instead of vector ones. Instead of parallelization it recommends some (in my opinion very ugly) hacks that enable you to take for example some control over what is stored in processor cache, how instructions are pipelined and how successful branch predictions will be. Those video software guys must be practicing black magic or something if they really get that 200% boost from MMX.

The problem I see here is that today's CPUs have too much internal logic that thinks it is smarter than you and reorders instructions, renames registers, controlls cache, ... I agree that's good for those 99.9% of the code that you don't want to bother optimizing, but for the few loops that need to be hand optimized, this is a nightmare. Instead of simply instructing the CPU to store this and that in its cache, I have to perform some weird sequence of reads from memory (take a look at some of the examples in that optimization guide if you don't believe me) so that I'll trick CPU's internal logic into thinking that perhaps it is a good idea after all to store that in the cache. And if I don't do it just right, the wrong things will end up in cache and that latest CPU will start emulating a 486. It would be nice if CPUs would have a mechanism to turn all this mess off and let me be in control for a while.

I miss the days when I could count the exact number of cycles the CPU needed to execute a function instead of having to measure it with 10% uncertainty.

I have to design, implement and document a simple electronic device that uses some kind of a microcontroller as a term project at the Faculty. Since I didn't find any of the proposed projects particularly interesting (why should I need another temperature controller or another digital thermometer?), I decided I will try to copy ThinkGeek's KeyKatcher.

It is critical for this device to be small (it should be handy, since you never know when you'll have to troubleshoot some software :) so I'm going to use a microcontroller that requires a minimum number of external components and a single high-density Flash or EEPROM chip.

I'm currently thinking of a combination of Atmel's AT89C2051 microcontroller and 24C1024 serial EEPROM. 2051 is a 8051-compatible chip that only needs an external quartz crystal for a clock generator. It has 2kb of internal read-only FlashROM for code, 128 bytes of RAM and runs at maximum clock frequency of 24 MHz, which should be more than enough for simple keylogging. 24C1024 would give my device the same memory capacity as the KeyKatcher, is both physically small (DIP8 or smaller package), has a large capacity and is simple to use.

FlashROM is cheaper then EEPROM and comes in even higher densities, but has two big problems: it mostly requires 3.3V or lower supply voltage (it's designed for cell-phones, etc. where power consumption is critical - keyboard supply voltage is 5V) and requires that a block of 128 or more bytes is written at once. Since 2051 only has 128 bytes of RAM, I don't have enough space there for a buffer that large. That means that I would have to either add some external RAM to 2051 or switch to a more powerfull CPU. First option means a bigger circuit and the second one means I would have to work with a CPU I don't know as well as 2051.

I guess 2051 won't be able to perform all of the functions KeyKatcher does (like searching through the stored data, recognizing URLs, etc.), but I don't think that should be a problem. You can always play with the captured keystrokes later once you download them to a PC.

Yesterday I spent five hours moving files from my old 40GB IBM DeathStar hard disk. After 4 years of service in my desktop it developed random bad sectors on the root and home partitions.

I found out that traditional Unix tools I used aren't really convenient for this task. I've read somewhere that the safest way to move a lot of data from one disk to the other is with tar like this:

$ cd /olddir
$ tar -c . | tar -C /newdir -x

This turned out to be a really bad idea. First, tar didn't restore hard links. I used hard links a lot in some hacks a while a ago and the copy of my home directory used far more disk space than the original. Second, tar has a weird idea of when to stop if it encounters a file it can't read. It will kind of continue after the error but won't copy everything it should. It will skip files that are perfectly OK.

The ordinary cp -a turned out to preserve hard links, but had the annoying property to stop at each corrupted file.

In the end the best solution I found was to use mv on each top level directory on the partition. It copied all files, skipped those that could not be read, but didn't delete anything from the source drive.

Today I've stumbled upon this sequence of commands that will allow you to unpack a Debian package even when dpkg isn't functional:

# cd /
# ar x xxx.deb
# tar xzf data.tar.gz

If I only knew this was so simple a few months ago. It would have saved me quite a few hours of unpacking packages on a functional machine and transferring files via a floppy to a crashed one.

Sometimes when I'm bored I apt-get random packages from the Debian archive on my computer. More than once I found some great piece of software this way that I didn't even know they existed (last two packages I remember finding were games Supertux and Viruskiller).

Now it looks like some people at Debian also realized that because of the sheer number of packages many nice packages remain unnoticed. A few days ago debaday.debian.org went on-line, which promises that it will present one Debian package a day. They even offer a RSS feed.

Anyone care to organize a Slovenian Gnome 2.12 release party tomorrow? Somehow I don't feel like traveling all the way to Madrid for this occasion.

OpenUsability is a project that brings Open Source Developers and Usability Experts together.

Quite a lot of free software projects seem to have registered on the site already. I was randomly browsing the site and the first forum post I looked at contained a link this interesting article about prototyping graphical user interfaces on paper. I'll definitely try this approach next time I will be writing a program with a GUI.

Just spent 30 minutes solving another problem that I could fix in 30 seconds if I would only get a proper error message. Whoever wrote OS X printing system obviously thought that error handling is for the weak. I just found out that OS X will simply say "Print job stopped" (in exactly the same way as if someone stopped the job manually) instead of reporting an error if lpd daemon on a remote printer refuses the connection.

After checking all sorts of configuration options in OS X, adding and removing printers I finally run tcpdump to see what exactly is happening on the network. This is what I got:

17:36:01.864288 IP orion.printer > sen.1012: P 1:57(56) ack 5 win 1448 <nop,nop,timestamp 20706385 3602102236>
        0x0000:  4500 006c d2e0 4000 4006 e450 c0a8 0102  E..l..@.@..P....
        0x0010:  c0a8 0108 0203 03f4 983c 26f2 d079 9481  .........<&..y..
        0x0020:  8018 05a8 7bbb 0000 0101 080a 013b f451  ....{........;.Q
        0x0030:  d6b3 b7dc 6f72 696f 6e3a 206c 7064 3a20  ....orion:.lpd:.
        0x0040:  596f 7572 2068 6f73 7420 646f 6573 206e  Your.host.does.n
        0x0050:  6f74 2068 6176 6520 6c69 6e65 2070 7269  ot.have.line.pri
        0x0060:  6e74 6572 2061 6363 6573 730a            nter.access.

Duh. One vi session of the /etc/hosts.lpd later, I could print again from my Powerbook, but how did OS X understand this as "Print job stopped" is beyond my comprehension.

The following piece of code has one simple syntax error on the first line:

287 for(x=0;x<XMAX/2+!;x++) {
288        for(y=0;y<YMAX/2+1;y++) {
289                for(z=ZMAX/2;z<ZMAX+1;z++) {
290                        space[x][y][z]=0;
291                }
292        }
293 }

When you try to compile it with GCC 4.0, you get this torrent of very informative error messages that point you to all other lines in the source file except the one that has the error. GCC obviously wants to make sure that the line with the error is the very last line of code that you will check.

lines.c:291: warning: control reaches end of non-void function
lines.c: At top level:
lines.c:324: warning: type defaults to ‘int’ in declaration of ‘id’
lines.c:324: error: initializer element is not constant
lines.c:324: warning: data definition has no type or storage class
lines.c:329: error: syntax error before ‘for’
lines.c:338: warning: type defaults to ‘int’ in declaration of ‘black’
lines.c:338: error: initializer element is not constant
lines.c:338: warning: data definition has no type or storage class
lines.c:340: error: syntax error before numeric constant
lines.c:340: warning: type defaults to ‘int’ in declaration of ‘g2_set_line_widt h’
lines.c:340: error: conflicting types for ‘g2_set_line_width’
/usr/include/g2.h:121: error: previous declaration of ‘g2_set_line_width’ was he re
lines.c:340: warning: data definition has no type or storage class
lines.c:342: warning: type defaults to ‘int’ in declaration of ‘s_draw’
lines.c:342: warning: parameter names (without types) in function declaration
lines.c:342: error: conflicting types for ‘s_draw’
lines.c:206: error: previous definition of ‘s_draw’ was here
lines.c:342: warning: data definition has no type or storage class
lines.c:344: warning: type defaults to ‘int’ in declaration of ‘g2_close’
lines.c:344: warning: parameter names (without types) in function declaration
lines.c:344: error: conflicting types for ‘g2_close’
/usr/include/g2.h:107: error: previous declaration of ‘g2_close’ was here
lines.c:344: warning: data definition has no type or storage class
lines.c:345: error: syntax error before string constant
lines.c:345: warning: type defaults to ‘int’ in declaration of ‘printf’
lines.c:345: error: conflicting types for ‘printf’
lines.c:345: note: a parameter list with an ellipsis can’t match an empty parame ter name list declaration
lines.c:345: warning: data definition has no type or storage class

Today I've found this review of cluster-oriented Linux distributions by a magazine called PC Quest.

They compared ClusterKnoppix, ParallelKnoppix and Tablix on Morphix. Unfortunately it seems that the authors of the article confused Tablix (software for solving timetabling problems) and Tablix on Morphix (slightly modified Morphix GNU/Linux distribution).

You can download the source code for ATC Advanced here.

If you want to try it, please keep in mind that most of this was written between 1am and 3am, so expect lots of weird code and strange error messages.

To compile, first edit common/pathnames.h and correct the hardcoded paths there (they should point to the location where you extracted the tar file). Then do a make clean && make. I haven't tried make install yet and I don't know what it does. Just run the binaries from the source tree for now.

To play you first have to start the server. Run it like this: server/atca_server -g OHare. You can replace "OHare" with any other game that is available in the games/ directory. After the server is running, start one or more clients like this: client/atca_client. Each client will ask you for your nickname and the IP of the server. After a few moments you should see the radar display.

All clients see the same radar screen. The only difference are the colors of the airplanes. Unassigned new airplanes are red. Green airplanes are assigned to you. White airplanes are assigned to other clients or are ignored (you can ignore an airplane by using the "i" command). You can assign an unassigned airplane by using the "p" command. Once the airplane has been assigned to someone, it can not be reassigned.

Please do not send bug reports unless you have a patch.

Here you can see the server running in the upper left window. I've removed all functions that were drawing the radar screen from the server code. Maybe I'll implement some kind of a command line in the future. Currently the server only outputs some status messages to the terminal.

Other three windows on the above screen shot are clients connected to the server. The lower left client just ordered plane "a" to change heading to 0 degrees. The upper right client ordered the same plane to turn to 90 degrees and the last client ordered airplane "b" to climb to 9000 feet.

I'm using the Simple Network Layer library for networking. It is very simple to use (contains only 4 functions) and as far as I can see it works perfectly. The only problem I've found is that I can't use telnet to directly talk to the server (the library returns "buffer allocation error") even though the network protocol I use is plain ASCII.

This is a screenshot of the first kind-of-working version of my multiplayer ATC hack.

ATC (Air Traffic Controller) is a simple game by Ed James which lets you try your hand at the nerve wracking duties of the air traffic controller without endangering the lives of millions of travelers each year. The original only supports one player and uses simple ncurses-based monochrome ASCII graphics.

Loopus and I got the idea at What The Hack to make a multiplayer version that could be played over the internet and perhaps use SDL for better graphics.

In the screenshot above, the top window is running the server. Any number of clients (one can be seen running in the bottom window) can then connect to the server and they can all send commands to all airplanes on the radar screen.

I'll release the source code as soon as I'll get this even remotely playable. I still have problems with sending commands over the network connection (the game uses a quite interesting state machine for entering commands for airplanes).

I'll post some more details tommorow.

If your tcpdump isn't showing any packets when you know it should be (like, you can ping another machine, but tcpdump doesn't show any packets on the wire), then the problem can be that it can't get a reverse DNS look up for the IPs in the packets it is receiving. You can use the -n option to turn off reverse lookups and solve this problem (or better, fix your DNS configuration).