Andreas Fuchs' Journal: November 2006 Archives

I don’t use FreeBSD anymore, but for some reason, this topic still interests me.

So it brings me great joy to see that NIIMI Satoshi has a patch that adds threading support to SBCL on FreeBSD! I tested it on FreeBSD 6.1-RELEASE running in Parallels Desktop for Mac, and all tests passed 2 of 3 times. I think this would be a good time for all FreeBSD-using lispers (especially those running it natively) to help us test this change.

Good hunting!

Xach writes:

I imagine they are writing […] to their future self, who will be impressed with how hip and with-it younger self was.

Xach knows me better than I know myself. This time, it’s because I do have another (not really top secret, but slightly embarassing, so not linked here) blog. Have had one for 5 years, in fact. And its sole purpose is to impress my future self with how hip and with-it my younger self was.

Somehow, it never has had the effect on my then-present selves that my then-past selves intended. Which, if I think about it, is a good thing.

As I own my own mail server, my mail setup is very much engineered to fit my own needs. This means everything works as I think it should, but it also means I have to hack things myself if I need them. This time, I needed a program that can move IMAP mails to archive mailboxes.

(Side note: My IMAP mail server is dovecot, which supports keeping mail in different namespaces in separate storage system. For current mail, Maildir is handy: needs no locking, and it’s hard to corrupt a mailbox. For archiving, mbox is better, because one mailbox takes up only one file.)

The plan was this: For every mail in “mail.lisp.sbcl.arch”, I want it to move the mail to the mail box “archive.<year>.lisp.sbcl.arch”. This would have been a tedious mark&drag operation in any gui-based mailer, and hopelessly slow in the emacs-based mailers (which support this operation) like wanderlust. Mel-base to the rescue!

I’ll keep it short: This code was the result. It requires mel-base (obviously), and I only tested it on sbcl.

So, this is how you use it:

1. Install mel-base.

2. Create a file passwords.lisp in the same directory as message-archiver.lisp, with contents like these:

       (setf *me* "username")
       (setf *my-pass* "password")
       (setf *my-host* "imap-server")
   

3. load message-archiver.lisp.
4. run (imap-archiver:archive-messages "lisp.phemlock") ; and it will move all mails in the mailbox “mail.lisp.phemlock” to the mailbox “archive.<year>.lisp.phemlock”, with <year> being the year in the message’s Date: header field.

But wait! This is very slow on current versions of mel-base (0.7-2)! Why? The code that copies a message from one folder to another works the same for all folder types: it reads the entire message from the server and sends it back again. As a side effect, this removes all marks. Ow. But we’re lucky that Jochen Schmidt is a good hacker and designed mel such that this is easily fixed.

Like everything else in mel-base, operations on a folder have their own protocol, and copying messages from a folder to another has its own generic function, copy-message-using-folders:

    (defgeneric COPY-MESSAGE-USING-FOLDERS (message message-folder sink-folder))

As it is, we can easily make a method that specializes on the case where we copy a message from an imap folder to another imap folder. We only need to check that they’re on the same server (for that, I assume they’re the same if the server name, port, user name and password are the same), and issue the correct UID COPY command to the imap server, and we’re good to go:

    (in-package :mel.folders.imap)
    (defmethod copy-message-using-folders :around ((message message) (message-folder imap-folder) (sink-folder imap-folder))
      "Copy a message between two imap folders. We can optimize this case if the folders are on the same server."
      (if (and (equal (host message-folder) (host sink-folder))
               (equal (username message-folder) (username sink-folder))
               (equal (password message-folder) (password sink-folder))
               (equal (imap-port message-folder) (imap-port sink-folder)))
          (progn
            (send-command message-folder "~A uid copy ~A ~A" "UID" (uid message) (mailbox sink-folder))
            (process-response message-folder :on-uid (lambda (m) m)))
          ;; if we're not using the same server, play it safe
          (call-next-method)))

Evaluate this, and suddenly everything is 10 times as fast, and marks are preserved after moving the message.

The lesson for today: Good libraries provide functionality that works well enough for the typical use case. Great libraries let you extend them to support your own use cases.

As a pleasant side-effect from CLISP boinkmarks availability, I can now make available a git-ified clone of CLISP’s CVS repository that is synced once every hour.

To use it, point your git or cogito client at git://sbcl.boinkor.net/clisp. If you want to browse the commits, tags and branches, you can get a gitweb summary (with rss!) here.

Of course, tips I posted about the git-ified SBCL on this blog before (like the one about finding bugs by using git bisect) apply to the git-ified CLISP as well!

I gave in and updated my lisp hacking-related emacs configuration today. These three snippets make my life easier now:

• Turning off the “unsafe local variable” warning for common lisp source files. Several packages use the -*- convention to set the major mode; some also set variables like Package, Base and Syntax. Emacs 22 doesn’t know that they’re harmless and prompts almost every time I open a new lisp file. Ugh. Use this:

  (put 'package 'safe-local-variable 'symbolp)
  (put 'Package 'safe-local-variable 'symbolp)
  (put 'syntax 'safe-local-variable 'symbolp)
  (put 'Syntax 'safe-local-variable 'symbolp)
  (put 'Base 'safe-local-variable 'integerp)
  (put 'base 'safe-local-variable 'integerp)
  

  and remove all those unnecessary safe-local-variable-values customizations.

• Wrapping the current defun in (eval-when (:compile-toplevel :load-toplevel :execute) …): I always misspell at least one of the keywords, so I bound C-c e to wrap the current defun (or the current + prefix-arg - 1 next defuns) in an eval-when:

  (defun asf-eval-whenify (&optional n)
    (interactive "*p")
    (save-excursion
      (if (and (boundp 'slime-repl-input-start-mark)
               slime-repl-input-start-mark)
          (slime-repl-beginning-of-defun)
          (beginning-of-defun))
      (paredit-wrap-sexp n)
      (insert "eval-when (:compile-toplevel :load-toplevel :execute)\n")
      (slime-reindent-defun)))
  (define-key lisp-mode-map [(control c) ?e] 'asf-eval-whenify)
  

  (this requires both Slime and Paredit)

• Fixing Command-left in Aquamacs: A-left (a.k.a. Command-left) goes to the beginning of the line. This is useful (and consistent with OS X) everywhere but in slime REPL mode. Unfortunately, shadowing the binding in the repl mode map isn’t possible, as macosx-mode-map’s A-left overrules that of the slime repl mode. Here’s a piece of advice to work around that:

  (when (fboundp 'beginning-of-visual-line)
    (defadvice beginning-of-visual-line (around slime-repl-bol first () activate)
      (if (eq major-mode 'slime-repl-mode)
          (call-interactively 'slime-repl-bol)
          ad-do-it)))
  

  Update: In Aquamacs 1.0b, you have to use:

  (when (fboundp 'visual-line-down)
    (defadvice beginning-of-line (around slime-repl-bol first () activate)
      (if (eq major-mode 'slime-repl-mode)
          (call-interactively 'slime-repl-bol)
          ad-do-it)))
  

I’ve spent the last few hours adding a bit of javascript/araneida magick to the boinkmarks web interface. If you have javascript enabled, and your browser is supported by dojo, it’s possible to select the hosts, implementations, releases that you want in a matter of seconds (and not in 2-3 minutes). Yay!

Also, that taught me some useful lessons:

• I’m not cut out for web development. Seriously, ew.
• Parenscript is nice. But then you find out you can’t write something.innerHTML = ..., because there is no way for it to emit a variable name that consists only in part of capital letters — and javascript is case-sensitive! Argh! You can get innerHTML by using (in lisp) something.inner-h-t-m-l. (thanks to Marco Baringer and Ivan Toshov)

In other news, Postgresql = love. It’s really a joy to search for performance problems with it. The last one I weeded out was this:

When you join a small table to a big table in order to limit the number of resulting rows to ones matched in a big table, don’t use JOIN, use WHERE EXISTS.

For example, if you want smalltable’s foo column for rows that are related to those rows in bigtable that match bigtable.condition=’something’, don’t use

select smalltable.id, smalltable.foo from smalltable join bigtable on bigtable.id=smalltable.id where bigtable.condition = 'something'

, use:

select id, foo from smalltable where exists (select * from bigtable where bigtable.id = smalltable.id and bigtable.condition = 'something' limit 1).

(And don’t forget to keep an index on (bigtable.id, bigtable.condition)!) Rewriting sql statements like this (and using an index) gave the queries in the boinkmarks web app a speed boost of anything between 30x and 200x.

It’s still slow, but at least now I can’t do anything obvious about it anymore. (-:

Here’s the explanation of yesterday’s cryptic announcement:

Prompted by a late-night discussion on #lisp, I added build/benchmark support for CLISP to autobench. It’s currently churning away at early releases, but some numbers are already in.

Graphs are available for AMD64 in 32-bit mode and AMD64 in native mode.

When everything is working correctly, I’ll start benchmarking CVS commits as they trickle in (just like I do for SBCL). If any volunteers want to dedicate hard disk space (the git tree is 186 MB, plus 1.5MB for every revision in the build archive) and processing power (building one revision takes around 10 minutes, 3 benchmark runs take 27 minutes), that would be much appreciated. See the updated autobench setup guide for details; if you need more convincing, I have a flyer that highlights the advantages of running boinkmarks for you.

…But to me, it spells “success”:

#<CLISP 2.41-2006-10-13-gea222e0>/(:ARCH :EMULATED-X86): Build:OK BM3:012

Results soon.

Hacked on the SSH Plugin some more. Now it can do:

• Connections to hosts that are not in the index. Type . (the period character) to enter Text mode, enter a host name and select the “Open SSH Connection” action.
• Connections to hosts with non-default user names. Use the “Open SSH Connection as User…” action for that.
• Oh, and there’s some documentation!

I’m declaring this feature-complete. If you installed the previous, pretest version, you will have to remove the “SSH Plugin” before installing the new one: Open Preferences, Plug-ins, select the SSH Plugin and use “Delete selected Plug-ins”.

Get the new version here: DMG. As always, there’s the source (MIT licenced) and git repository, too.

Troels Henriksen has made available his modifications to McCLIM: He added an input editor (named “DREI”) that is based on climacs. The greater plan seems to be to make Climacs use that input editor for all its editing panes.

Apart from less code duplication and embeddability of a really nice emacs in every clim application that uses a text-editor gadget, this adds a few cool things to the input editing in current clim applications. This screen shot

shows:

• Lisp syntax highlighting,
• (almost) correct indentation of forms, and
• nicely formatted “noise” strings: (pathname).

In a recent entry, I compared deskbar-applet with Quicksilver and introduced a plugin I had written. Well, I now own a macintosh, and so I wanted Quicksilver to gracefully handle SSH connections, too.

Of course, it’s possible to make a safari/mozilla/omniweb bookmark of the form ssh://some-host/ and have quicksilver add them to its index, but that gets old quickly. I want all hosts in my known_hosts file to be available, and I don’t want to type in all 23 of them - again. So I had to write my own SSH plugin, which turned out to be fun.

Quicksilver is pretty badly documented, but the kind people on #quicksilver were very helpful — thanks, ytrewq2! — and so I managed to create a working plugin.

Here’s a pretest version that reads in known_hosts (user and system-wide) and the ssh client config and makes them available as ssh URLs. I have some bigger plans with that, but for now this works well enough for me: updated DMG (see), source (MIT licenced), git repository

Incidentally, this was my first contact with Objective C. It feels like a nice language, and the development environment (Xcode) is pretty awesome. There’s very high-quality documentation for everything included — except Quicksilver internals, of course. (-:

I’ve announced the git repository a few weeks ago. Here’s something very nice you can do with it: Run a binary search on revisions to find out the version of SBCL that caused a bug. This helps enormously when searching for the cause of bugs.

You can use the CVS repository for this, too, but it doesn’t include any of the nice functions I mention in this tutorial; you need a start time and an end time, and do month/day/hour/minute calculations yourself. Git allows you to bisect based on changesets, and it includes a nice graphical representation of bisection progress.

Here’s how you do it:

1. Make an automated test case that you can run from a script. This may seem optional, but it gets tiresome (and error-prone) if you test manually.

2. Check out the sbcl tree: git clone git://sbcl.boinkor.net/sbcl

3. Change to the directory, look at the revision log: git log

   The lines starting with “commit” contain sha-1 commit IDs. A few lines after that, you see the log summary containing the sbcl revision number.

4. Find the version that you know is broken, and a version that you know works and copy their commit IDs. I’ll call the good version’s commit ID 6584a2c88efaa6931083721adae2f9f10e0fefd5 and the bad one’s commit ID 1de12891f900d156ed035a097561ecd7755a256a.

5. start bisection search: git bisect start

6. Now mark the good and bad revision:

   • git bisect good 6584a2c88efaa6931083721adae2f9f10e0fefd5
   • git bisect bad 1de12891f900d156ed035a097561ecd7755a256a

   This will give you output similar to:

   Bisecting:        9 revisions left to test after this
   [90a83478829f33b91f6300c183b374a968bc13c6] 0.9.18.20: correct step-frame logic on non-x86oids

   From now on, after every step, you can look at a first bisection history in gitk: git bisect visualize

7. Start the actual bisection: git bisect next

   This gives you:

   Bisecting:        9 revisions left to test after this
   [90a83478829f33b91f6300c183b374a968bc13c6] 0.9.18.20: correct step-frame logic on non-x86oids
   $ cat version.lisp-expr
   [...]
   "0.9.18.20"

8. Run the test script.

   • If it fails, use git bisect bad to mark the current revision as bad,
   • if it succeeds, use git bisect good to mark it as good.

   You’ll get something like:

   Bisecting:        5 revisions left to test after this
   [7bad074650949dc5427711b93ff615d9c17308d9] 0.9.18.25:

   As mentioned above, you can always view the bisection history using git bisect visualize.

9. Repeat steps 7 and 8, until there are no more versions to bisect. After running the final test and marking the version with git bisect good or git bisect bad, You’ll get something like the following output:

   a30a3d82293eca3eb036ea0c713b0da31c0467dc is first bad commit
   commit a30a3d82293eca3eb036ea0c713b0da31c0467dc
   Author: Nikodemus Siivola <nikodemus@random-state.net>
   Date:   Thu Nov 2 15:11:25 2006 +0000
   0.9.18.27: fix darwin build
    [...]

And that’s it! You identified the broken revision. When reporting bugs to the sbcl mailing list, please include the version number (in the example above, that would be 0.9.18.27).

If you want to cancel or pause bisection at any time, you can use the following commands:

• git bisect log > bisection-logfile # (optionally) save the current progress
• git bisect reset # reset bisection to the revision where you started

To resume bisection from a log file, use git bisect replay bisection-logfile.

I hope this guide helps you identify bugs quickly. Good hunting!

We released McCLIM 0.9.3, code-named “All Souls’ Day” today. Most prominently, this release:

• Plugs a horrible memory leak in incremental redisplay, along with a few optimizations.
• Has a new backend, gtkairo (uses GTK for gadgets, and Cairo for rendering)
• Includes lots and lots of bug fixes and new features.

The “haha, this is embarrassing” department mentions that my brain misfired while sending the release announcement. I wrote “All Saints’ Day” instead of “All Souls’ Day”. This is now corrected.