Archive for the 'Programming' Category

Note: I've reorganized this site to use tags; the category archive remains to support old links. Only posts prior to April, 2006 are categorized. Tag Archive »

Wisdom of the Documentation

Today, I spent some time staring at an old piece of code that I had written at least a year ago. It’s been in testing several times, but never put into production (the project it is tied to has been bumped on several occaisions). Today, it was back in testing.

The code is a failry simple web service, written in Perl. I like Perl. I have no illusions that I’m a fantastic Perl hacker, but I know the language well, though both experience and reading. I’ve read most of the O’Reilly Perl titles, including Programming Perl (“The Camel”), which I’ve read cover to cover at least three times. I still find myself looking things up, usually to refresh my memory about something I can remember reading, or some syntax detail I can’t get right (one of the perils of working in multiple languages). At least I generally know where to look.

So this web service has been tested before. It works in a browser, and it works when called by my test client. It’s been tested with a third-party bit of code. Today, it was tested by Dave, using some custom client code he had written in C#. And it worked… if he told his http library to ignore HTTP protocol errors. If he didn’t, his library complained.

And so I stared at the code for a while. By coincidence, I’d been reading the HTTP Spec over the weekend (yes, I’m a geek), and was pretty sure my response was good- a bare-minumum response, along the lines of:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8

Single Line Response

I double checked the spec anyway, and kept staring at the code. I was about to start grasping at straws and adding additional entity headers to the response (such as Content-Length), when I finally stared at the code long enough. I saw something like this:

print "HTTP/1.1 $status\n"
print "Content-Type: text/plain; charset=utf-8\n"

Then it hit me- "\n" in perl is a “magic newline”- it conforms to the newline convention on the system in question. HTTP, on the other hand, requires ASCII CR+LF (Cariage Return + Line Feed, or "\r\n" in C) as a line terminator. Apparently all of the code thrown at the service before today was a bit forgiving. I changed the strings to send CRLF using octal escape sequences ("\015\012"), and everything was fine. I was a bit ticked about the mistake… I new both the HTTP requirement for CRLF and the Perl treatment of "\n" when I originally wrote the code; it was a dumb mistake. Also aggravating that it took so long to spot.

And there my tale should end. But this evening, I started wondering if the octal sequence was the most Perlish way to send a CRLF. I knew that "\r" is fine for the CR, but you can’t use "\n" for the LF – it’s magic in perl, and behaves differently on different platforms. I began to wonder if Perl has a backslash-escape for LF that is always LF. Eventually, I had to check for myself, so I referred to the Quote and Quote-like Operators section of the perlop man page. (Sadly, I knew right where to look, right down to the name of the section. Geek, remember?)

Turns out the manpage specifically recommends the octal form for networking applications (at least I got that right), but then it twists the knife:

If you get in the habit of using "\n" for networking, you may be burned some day.

D’oh.

HOWTO See what’s changed in the file you’re editing in Emacs

Warning: Geek threshold exceeded. If you don’t know what Emacs is, this post will mean nothing to you.

I use Emacs as my primary editor these days, and I tend to have lots of buffers open at once. Every so often, I’ll go to close Emacs or just close some buffers, only to be alerted “Buffer XYZ modified; kill anyway? (y or n).

What? I opened that buffer 3 days ago. I don’t know if I should save those changes… what changed? Now, my copy of XEmacs has ediff, a very nice interactive diff tool. You can diff two buffers, two files, three files, buffers against revisions (if the file is under source code control), etc. What you can’t do is diff a buffer against the underlying file on the file system.

Now, I could save the buffer in question to a temporary location, and then ediff that against the original. But where’s the elegance in that? I wanted a better solution, so I asked Google.

I found the answer (well, most of it) in a 1996 post to comp.emacs by Larry Rosenberg. He offered a function for doing a quick diff of the current buffer against the underlying file system version. I bound this C-c d in emacs. I then expanded it just a bit. I often use context-diffs (diff -c), so I wanted the option, but for long lists of diffs, sometimes it’s just too much. So, I made it an option of the command. Invoked normally, it shows a plain diff. When prefixed with C-u (using my binding, this becomes C-u C-c d), it runs a context diff.

Just to be clear – Larry did all the real work back in 1996. But considering my understanding of elisp is only slightly better than my understanding of Sandskrit, I’m pleased with my modification.

Here’s the function definition, along with the keybinding, from my emacs init file:

(defun diff-buffer-against-file (context)
    "diff the current [edited] buffer and the file of the same name"
    (interactive "P")
    (let (  ($file buffer-file-name)
            ($tempFile "/tmp/emacs.diff")
            ($tempBuffer "emacs.diff"))
        (delete-other-windows)
        (push-mark (point) t)
        (generate-new-buffer $tempFile)
        (copy-to-buffer $tempBuffer (point-min) (point-max))
        (set-buffer $tempBuffer)
        (write-file $tempFile)
        (shell-command (concat (if context "diff -c " "diff ") $file " " $tempFile))
        (kill-buffer $tempFile)
        (pop-mark)
    )
)

(global-set-key "\C-cd" 'diff-buffer-against-file)

Stupid Perl Debugger Tricks

I’ve been playing with Python alot lately in my spare time, but I still use mostly Perl at work. One of the handy things about Python is the interactive mode; I like it so much I even cloned it for Perl some time ago. Even without my Perlthon script, you can get a quick approximation in perl using the perl debugger and a command-line script:

perl -de1

(That’s a one, not an el.) The above will invoke perl with the debugger (-d), debugging a very simple script (-e1, which is to say -e '1;'). Once the debugger starts, you can just type perl statements, and can use x <expr> to inspect values.

Whichever way you play with interactive Perl, testing regular expressions can be a pain. It’s not too bad under Python, given Python’s use of match objects:

import re
re.search('regex', 'string').group(0)

The second line runs the regex against the string, and prints the entire match. If your regex doesn’t match at all, you get an error, but that’s fine (and self explanatory). If your regex doesn’t perform as expected, repeated attempts make it easy to triangulate. If you have Python’s readline module installed, you can just hit UpArrow after the test, tweak the regex, lather, rinse, repeat. I wanted the same flexibility with interactive Perl; it turns out to be trivial:

# perl debugger version
x ('string' =~ /regex/, $&)[-1]

# perlthon version
('string' =~ /regex/, $&)[-1];

The regex match operation will return a list of matched groups, which can be handy at times. For testing a complicated regex, I often just want to see the whole match to be sure I’m getting what I expect. The array notation accomplishes this nicely.

Python unittest Greenbar Without the GUI

Lately I’ve been learning Python (again), and trying out Unit Testing using Python’s unittest module. In many unit testing tools, each time the test suite is run, a progress bar is shown, which is green as long as the tests pass, but turns red when a test fails. Passing your tests is called “getting a greenbar,” and is the mark of success.

I’ve been working on my Powerbook, using the default Python install. I work with two terminal windows open; one running Emacs, and the other to run my tests in. The unittest module provides a command line method of running all tests, with output appearing on the terminal. I wanted to have a greenbar/redbar indicator, but none is present.

There is a GUI version, unittestgui, available which uses tkinter, but doesn’t ship with OS X. Google turned up a Cocoa app for OS X for running Python unit tests, but it hasn’t been updated in a few years, and just crashes on my Powerbook.

Since I prefer to develop in a terminal session anyway, I decided I wanted a way to run my unit tests from the shell, and see the red/green effect right in the terminal. Thus I created utest, a Bash script that runs a unittest, and changes the display text color using ANSI escape codes based on the output of the test. Since unittest prints all testing output to STDERR, I redirect it to STDOUT and pipe the the whole shooting match into perl one-liner that parrots eveything, while setting colors as appropriate. Output starts green, but turns red if the output contains FAIL or ERROR. It also unbuffers standard out, so the results appear correctly.

Here it is in action, when all tests pass:

utest with passed tests

And here it is, with a failing test:

utest with failed test

If you can’t see these images, you’re probably using IE, so don’t come crying to me. You know what to do. If you’re wondering about my tabbed terminal, it’s iTerm.

Hackers and Painters

Via Tim Bray, via Tim Bray, via, well, you know: Go and read Hackers and Painters, an essay by Paul Graham. This is a must-read for anyone who considers themselves a hacker, especially those like me who make a living at it. A few choice quotes:

I tended to just spew out code that was hopelessly broken, and gradually beat it into shape. Debugging, I was taught, was a kind of final pass where you caught typos and oversights. The way I worked, it seemed like programming consisted of debugging.

For a long time I felt bad about this, just as I once felt bad that I didn’t hold my pencil the way they taught me to in elementary school. If I had only looked over at the other makers, the painters or the architects, I would have realized that there was a name for what I was doing: sketching. As far as I can tell, the way they taught me to program in college was all wrong. You should figure out programs as you’re writing them, just as writers and painters and architects do.

And also:

In hacking, like painting, work comes in cycles. Sometimes you get excited about some new project and you want to work sixteen hours a day on it. Other times nothing seems interesting.

Amen, Brother.