My work is the finger work

Stars (and brackets) on thars

2012-02-09T00:00:00-08:00

One of my colleagues recently committed a change with the comment:

Log Message:
-----------
Switching from "type *varname" to "type* varname"

It gave me hope for the future, that we might overcome the legacy of K&R’s terrible mistake when they pioneered the styles foo *pointer and foo array[].

The world has mostly come around to the view that [] is part of the type, not the variable (i.e. foo[] is how you declare ‘array of foo’). In spite of this enlightenment, people cling to the syntax foo *pointer even though obviously foo* is how you declare ‘pointer to foo’.

The reasoning is that you can declare:

foo *pointerToFoo, stackAllocatedFoo;

which is plainly confusing and a bad idea.

Gosling was originally under the spell of K&R, but eventually came to his senses. Unfortunately, in Java the legacy of that madness lives on:

public class Test {
    public static void main (String[] args) {
        String[] one = {}, two = {}, three[] = {};
        // one and two are of type String[], three is of type String[][]
    }
}

Naturally, C clings firmly to the past. The following is perfectly legal, if not the most maintainable, C:

#include <stdio.h>

int main (int argc, char** argv) {
    char* foo = 0, bar = 0;
    printf("%ld %ld\n", sizeof(foo), sizeof(bar));
    return 0;
}

// prints 8 1 (or 4 1 if you're on ye olde 32-bit machine)

The perspective of the enlightened language designer is that special-purpose type modifiers like * and [] are a bad idea.

Arrays can be handled simply. They should be a parameterized type. Scala and Haskell (and probably other Haskell-influenced languages) get this right:

val one :Array[String] = { "foo", "bar" };

No need for special syntax. In Scala’s case they’ve disallowed the declaration of multiple variables in the same clause, except for this weird construct which is a concession made to support Scala’s highly unfortunate enum pattern:

val onePlusOne, twoTimesOne, fourDivTwo :Int = 2; // all vals bound to 2

Pointers are trickier business. For one, a civilized language doesn’t have pointers, which cuts the conversion pretty short. However, civilized languages do have have reference types, which are basically the same thing without the promise of naughtiness and adventure.

One can mostly get by with glossing over the difference between reference types and value types. When you write:

String foo = "bar";
int bar = 0xf00;

it’s not a great mystery that foo is a pointer to a string and bar is just an int on the stack. However, if you allow the creation of value types that are larger than the machine’s register size (e.g. Vector3, Matrix4, etc.), you will invariably want to pass them by reference.

Java cleverly avoids this conundrum by restricting itself to only built-in value types which fit into registers. That’s the sort of approach that’s likely to get you zero credit on your math homework.

C# takes a crack at a non-zero solution with ref parameters. You can declare:

void invert (ref Matrix4 matrix) { ... }

and your calls to invert will not involve copying 16 doubles. We can deduce Hejlsberg’s enlightened status by the fact that it’s ref Matrix4 and not Matrix4 ref (the latter being the moral equivalent of Matrix4*).

However, I expect that if the JVM some day supports value types, Odersky’s modeling of them in Scala will be something even more enlightened, like:

void invert (matrix :Ref[Matrix4]) { ... }

Such an approach nicely mirrors Array[Matrix4] and avoids the need to introduce a new keyword into the language. Scala already supports implicit conversions, so a conversion from A to Ref[A] is as simple as:

implict toRef[A <: AnyVal] (value :A) :Ref[A] = null // implemented by compiler magic!

Under the hood, the compiler would emit the appropriate byte codes to indicate by-reference-ness, but there’s no need to sully the language with evidence of those machinations when there are already perfectly good abstractions available to model them.

C# also supports out parameters, allowing one to write:

bool getId (out int id) { ... }

int id;
if (getId(id)) {
  // I have an id!
}

id is considered uninitialized in the body of getId and must be initialized before getId returns (modulo exceptions).

I’m not sure such a thing could be modeled with a standard OOP type system, so perhaps this is a good case for special syntax. However, this whole mechanism exists solely to support multiple return values. The combination of a Tuple value type, deconstructing binding, and under-the-hood optimizations solve this problem without special cases. The following is already legal Scala code:

def getId() :(Int,Boolean) = ...

val (id, gotId) = getId()
if (gotId) {
  // I have an id!
}

Just add value types to the JVM and said under-the-hood optimizations, and you’re good to go.

Anyhow, I seem to have rambled pretty far afield. I’ll stop philosophizing and go back to being happy that the world contains a few fewer type *foo declarations. Some day we may live in a world where there are no *s upon ours.

Happy New Year to Me!

2012-01-03T00:00:00-08:00

I logged into the server that hosts samskivert.com yesterday and noticed that someone had done a very ham fisted job of hacking into my server (in addition to whatever nefarious business they hacked in to perpetrate, they changed the uid of my primary account to 0 and left the passwd file thusly modified). That’s what I get for having anything other than Apache and sshd running.

Naturally, the only thing to do in such a case was to take off and nuke the site from orbit. So I slurped down all the data from my slice and pressed the big red button that wiped the slice and reinstalled a fresh Ubuntu image (upgrading everything in the process, how nice).

Then I copied all the data back up to the machine and started setting things back up. That’s when I realized that I forgot to grab the MySQL databases. Oops! No problem, I have scripts that back everything up to S3. I fired up the S3 console and discovered that the directory that should contain the database backups was empty. Double oops! I guess something broke at some point in the five years since I set all that up.

The only thing that used MySQL was my WordPress install. There went thirteen years of blogging. I was ready to declare the world a better place for lack of my inane writing, but then I realized that Google Reader probably had most of my blog archived. Not only that, but it made it pretty easy to slurp down said archives.

One of the things I do to simplify my life, now that I’m old and forgetful, is to give up on trying to re-memorize things that I’ve forgotten, and instead to change the world to match what little I can remember. For example, if I forget the password to some account and have to reset it, I change it to the first thing I guessed might be my password when I was trying to log in. In this case, since I forgot that I had MySQL databases that needed copying, I should probably avoid creating new databases that I will undoubtedly forget about the next time this sort of thing happens.

Instead of setting WordPress back up, I set up Jekyll: a bunch of Ruby scripts that generate a blog based on flat files. The flat files can be committed to a Git repository. I definitely did not forget to copy my Git repositories. Indeed, I went one better and just moved my website Git repository to Github so that I can not worry about backing it up in the first place.

A few Perl and Scala scripts later and I had converted the Google Reader blog archives into a format acceptable to Jekyll. It turns out the Google Reader archives only went back to 2005 or so. Having come this far, I didn’t feel like settling for recovering only half of my blog, so I slurped down the Google caches for my web site and extracted the posts from 1999 through 2005.

I took the opportunity to factor my reviews from my blog-proper (even though both contain half-baked opinions). I set up redirects that should allow RSS readers to seamlessly continue to operate with the new setup, though this post will put the test to that hypothesis.

Another nice thing about switching away from WordPress is that I no longer need to maintain fiddly WordPress caching plugins (and associated Apache .htaccess machinery), nor do I need to run a Varnish reverse proxy to ensure that things stay performant if I ever post something that the world wants to read. Simple simple simple!

“Fun with edge cases” or “Why your spec can never possibly be complete”

2011-08-12T00:00:00-07:00

(Apologies for the duplication with Google+, still sorting out my syndication process.)

What do you think should happen if you do the following in Java?

Map<String,String> map = new HashMap<String,String>();
map.put("foo", "bar");
Iterator<Map.Entry<String,String>> iter = map.entrySet().iterator();
Map.Entry<String,String> entry = iter.next();
iter.remove();
entry.setValue("baz");

Should the entry freak out because you tried to update the mapping after it has already been removed? Should it reintroduce a mapping from “foo” to “baz”? Should the call succeed but have no effect on the map?

Java actually specifies the behavior as undefined. So perhaps your spec can be complete by pushing the burden onto the developer of ensuring that they never rely on any undefined behavior. The HashMap implementation’s interpretation of “undefined” is “allow the call to succeed and don’t modify the underlying map.” That makes it more difficult to ensure you’re not relying on undefined behavior than if it, say, threw an exception.

Perhaps the question on your mind is “Gee Mike, how do you find yourself in the position of caring about something like this?” The answer to that lies in this code, which chooses a different interpretation of undefined.

Forecast: cloudy

2011-06-04T00:00:00-07:00

Far too often, Google App Engine reports that everything is smooth sailing:

when the view from my app looks substantially less rosy:

My app is not complex. It stuffs data into the app engine data store and pulls it back out again. The total size of all my entities is one meeeellion bytes. The code is not massive; it’s seven jar files, the largest of which is the app engine API. It has less than five regular users. Aside from the fact that GAE’s design necessitates that about 80% of the requests involve starting up a JVM and initializing the whole webapp, it really ought not to be taxing their gigascale infrastructure.

After being unable to use it for 48 hours — again — this broke-back camel is going to limp off and rewrite the backend in SQL and move the damned thing to my Slicehost server. Hopefully GAE will be working again soon so that I can download my data.

Too much Maven Kool-aid

2011-06-02T00:00:00-07:00

A post on the Sonatype blog cheerfully reports:

Adding an additional source folder is as simple as:

<project>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>1.1</version>
        <executions>
          <execution>
            <id>add-source</id>
            <phase>generate-sources</phase>
            <goals>
              <goal>add-source</goal>
            </goals>
            <configuration>
              <sources>
                <source>some directory</source>
                ...
              </sources>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

If that’s simple, I don’t want to see complicated.

How I Learned to Stop Worrying and Love (Coding on) the Mac OS

2011-05-24T00:00:00-07:00

I’ve been using Linux and X Windows as my primary operating system for development for a long time. Owing to the spotty or non-existent support of consumer applications over the years, I have generally maintained an auxiliary machine (sometimes Windows, sometimes a Mac) on which to run things like iTunes or the Flash Authoring Tool (or even the Flash player when Linux support was crap).

My primary machine has generally been a Mac, because I like the hardware, I just haven’t ever run Mac OS on it. Or rather, I would periodically attempt to switch to the Mac OS for development and then switch back to Linux after a week or so, annoyed at the hundred tiny ways in which my efficiency was reduced.

A few months ago, I resolved to overcome the remaining obstacles by whatever means necessary (even if it meant writing Application Enhancer modules, though it didn’t come to that). I’ve now been happily coding away on Mac OS for multiple months, so I think it’s time to declare victory and record for posterity the various hacks and configuration tweaks that make my life tolerable.

Virtual Desktops

The Mac has long lacked civilized virtual desktop support. Frankly, it still sucks, but between some hackery and some finger retraining, I’ve managed to cobble together behavior that I can live with. In the Linux world, I had what I consider to be the ideal behavior:

Instant switching between desktops. I don’t need to waste hundreds of milliseconds waiting for some animated transition to complete.
Direct switching to a desktop via Alt-Fn. Don’t make me use arrow keys to navigate my desktops (or heaven forbid, the mouse). I know where I want to go, and I want to get there with a single key combination. I also don’t want that key combination to be Alt-n because certain web browsers (i.e. Firefox and Chrome) use those keys to instantly switch between tabs, which I also use heavily.
When on a particular desktop, Alt-Tab switches between only the windows on that desktop. As far as I’m concerned, the other desktops don’t exist unless I switch to them. Each desktop is its own tiny universe.
When switching to a desktop, the window that had focus when I left that desktop has focus when I return.

I was not able to recreate this desktop utopia fully, but here’s what I did. First, disable desktop animation transitions thusly:

defaults write com.apple.dock workspaces-swoosh-animation-off -bool YES && killall Dock

Next, install Hyperspaces which allows you to map arbitrary key shortcuts for switching between spaces. You may also need to disable the default Spaces key shortcuts. Note also: the use of Hyperspaces works around an infuriating Spaces bug, which I repeatedly encountered when I was trying the “just give up on the utility of Alt-n in Firefox/Chrome” approach.

I still have to live with the fact that Alt-Tab and Spaces interact in absurd ways, and that Alt-Tab in general switches between applications, not windows. I have to switch between Alt-Tab and Alt-` if I want to get to a particular Terminal window. Spaces also does suboptimal things with focus when returning to a Space. It attempts to preserve focus in the same application if it can. If I have Terminal focused on one Space and switch to another Space with a Terminal window on it, that Terminal window will get focus regardless of what window last had focus on that Space. Only if no window from the same application is open on the target Space will it give focus to the last window that had it. Useful!

Ctrl-Meta-Cokebottle

I’m an emacs user, and thus I have many neurons in my peripheral nervous system dedicated to the use of the Control and Meta keys. (Meta is called Alt on PC10x keyboards and Command on Mac keyboards.) The way my fingers wish to use these keys causes numerous problems on the Mac OS.

I can’t use the sexy slim Mac keyboards, because I have to remap Caps-Lock to Ctrl (naturally), and the Caps-Lock key sucks. Five times out of ten, instead of registering ‘Ctrl-a’, the keyboard registers ‘a’ and happily inserts it into my buffer (same goes for ‘Ctrl-e’ and ‘e’, etc.). Since I use Ctrl-a/e about fifteen times a minute, this is somewhat problematic. Thus, in front of my sexy Apple Cinema display, above my sculpted aluminum Mac Pro, I have a very non-slim, boxy, black KeyTronic KT800U.

In addition to remapping Caps-Lock to Ctrl, I have to swap Command (Alt) and Option (funny-Windows-symbol); for some reason Apple chose to wire those two keys backwards on their keyboards. This is easily accomplished in the Modifier Keys popup of the Keyboard preferences, but it does not solve the larger problem that the Mac uses the Command key for all of its application shortcuts. In certain applications where I expect my emacs key bindings to work, like Terminal, it gets pretty annoying that, for example, Command-d splits the Terminal window into two panes, rather than deleting to the next word boundary. It’s not 15 times a minute annoying, but it’s still pretty bad.

This Terminal Command key issue had long been the primary insurmountable obstacle that sent me back to Linux. In Emacs.app, they thankfully make the Command key do the right thing, but I spend enough time in Terminal to eventually be driven crazy. Enter cmd-key-happy, my savior. This handy utility allows one to reclaim the Command key in any application, while leaving any specific Command key shortcuts in place (like Command-x/c/v and Command-Tab). It’s very useful if you want to use emacs key bindings in Eclipse or XCode as well.

I also went into the Keyboard → Keyboard Shortcuts preferences pane and disabled a bunch of global key shortcuts that caused my emacs-trained fingers to betray me in various circumstances. Specifically: Dashboard & Dock → Turn Dock Hiding On/Off, Front Row → Hide and show Front Row, Service → Make New Sticky Note, Application Shortcuts → Show Help menu.

I can has Terminal

I used keylaunch on Linux to provide global key shortcuts for opening a new xterm and a few other incidentals. Most of that I have learned to live without, but opening a new Terminal window to do some quick shell noodling is something I do all the damned time. Through a combination of Automator.app and some AppleScript, I managed to recreate this functionality without having to install yet another piece-of-crap shareware utility.

In Automator, I created a new Service automation, with the following Run AppleScript action:

on run {input, parameters}
	tell application "Terminal"
		do script ""
		activate
	end tell
	return input
end run

do script "" instructs Terminal to execute the empty command, and it opens a new Terminal in the current Space in which to do so. activate tells it to give said window focus. Since AppleScript has approximately fuckall understanding of Spaces, the fact that this works at all was a lucky happenstance.

This automation I then saved (cleverly named “New Terminal”). In the Keyboard → Keyboard Shortcuts preferences, under Application Shortcuts, I created a new shortcut (named “New Terminal”, which is how it knows which automation to use), and configured my desired key combination (Command-Esc). Said shortcut magically appears in the Services menu of every application, and the key combination usually even works when I press it (like many aspects of Mac OS, sometimes it mysteriously fails).

An Emacs on Every Desktop

Prior to setting up scripts to run one emacs instance on each virtual desktop, I coped with the default behavior, which was to open a new instance of emacs every time I ran emacs somefile from the command line. This wasn’t so bad, since I usually just open files from within my existing emacs instance, and it at least allowed me to have one (or sometimes more than one) emacs on each virtual desktop.

However, the default behavior on the Mac is for there to be one instance of Emacs.app running ever, and for all new files to open in that instance. This interacts very poorly with my normal workflow of having different virtual desktops devoted to different programming projects. For a while, I tried opening multiple frames (via M-x new-frame), and having one frame on each Space. Emacs.app was usually smart enough to respond to open -a Emacs.app somefile by opening the file in the frame on the current Space. The major downside was that all my buffers across all projects were all mixed together in a single emacs instance, which sucked in various ways.

After a great deal of digging, I discovered a terrible hack for determining the number of the current Space via AppleScript. I put that into a script that I could use with my previous setup to achieve a separate emacs instance on each Space.

#!/usr/bin/osascript

set xxVar to 0
tell application "System Events"
tell process "SystemUIServer"
set xVar to value of attribute "AXChildren" of menu bar 1
set cVar to count of xVar
repeat with iVar from 1 to cVar
    set zVar to value of attribute "AXDescription" of item iVar of xVar
    try
        if zVar = "spaces menu extra" then
            set xxVar to iVar
            exit repeat
        end if
    end try
end repeat
end tell
end tell

if xxVar = 0 then
    display dialog "Spaces Menu Extra not installed"
else
    tell application "System Events"
    tell process "SystemUIServer"
    set theCurrentSpace to value of menu bar item xxVar of menu bar 1
    end tell
    end tell

    -- Do what you want with "theCurrentSpace" variable starting here
    get theCurrentSpace as number
    -- Do what you want with "theCurrentSpace" variable ending here
end if

This script requires that the Spaces menu be visible, which duplicates the Hyperspaces menu (which cannot be disabled). I also end up with three or four Emacs.app icons in the dock and in the Alt-Tab menu (with no way to tell which is on which space, since they are reordered based on when they were last used). However, these are small prices to pay for the joy of segregated buffers.

Fine Beverages

Naturally, there are a few dozen Unix utilities that I needed in order to pursue my nefarious goals. In the past, I had used Fink or MacPorts, but now I’m a fan of Homebrew. I like not installing packages as root, and having them each installed into their own little directory tree and symlinked into the giant mishmash of /usr/local.

Your Suggestions Here

That’s the extent of the wrangling I’ve done to make the Mac OS a moderately pleasant place to develop. Well, that and to install an SSD because the filesystem or caching subsystem or something was causing builds to take 20-30% longer. If you, dear reader, have further suggestions for a reformed Linux user, please post them in the comments.

Euler 66

2011-01-14T00:00:00-08:00

Problem 066: (source):

import scala.collection.mutable.{Map => MMap}
object Euler066 extends EulerApp {
  def leastx (d :Int) = {
    val amap = MMap[Int,BigInt]()
    val pmap = MMap[Int,BigInt]()
    val qmap = MMap[Int,BigInt]()
    val ppmap = MMap[Int,BigInt]()
    val qqmap = MMap[Int,BigInt]()

    def a (i :Int) :BigInt = amap.getOrElseUpdate(i, i match {
      case 0 => math.sqrt(d).toInt
      case _ => (a(0) + pp(i))/qq(i)
    })

    def p (i :Int) :BigInt = pmap.getOrElseUpdate(i, i match {
      case 0 => a(0)
      case 1 => a(0)*a(1) + 1
      case _ => a(i)*p(i-1) + p(i-2)
    })

    def q (i :Int) :BigInt = qmap.getOrElseUpdate(i, i match {
      case 0 => 1
      case 1 => a(1)
      case _ => a(i)*q(i-1) + q(i-2)
    })

    def pp (i :Int) :BigInt = ppmap.getOrElseUpdate(i, i match {
      case 0 => 0
      case 1 => a(0)
      case _ => a(i-1)*qq(i-1) - pp(i-1)
    })

    def qq (i :Int) :BigInt = qqmap.getOrElseUpdate(i, i match {
      case 0 => 1
      case 1 => BigInt(d) - a(0)*a(0)
      case _ => (BigInt(d) - pp(i)*pp(i))/qq(i-1)
    })

    val as = Stream.from(0) map(a)
    val r = (as.tail takeWhile(ai => ai != as.head*2) length)
    if (r % 2 == 1) p(r) else p(2*r+1)
  }

  def square (i :Int) = i*i
  def issquare (d :Int) = square(math.sqrt(d).toInt) == d
  def answer = (2 to 1000 filterNot(issquare) map(d => (d, leastx(d))) sortBy(_._2) last)._1
}

My efforts to post my Euler solutions with commentary remain woefully behind (I just finished problem 110), but when I took action to close this gap, I discovered that I had skipped over problem 66 so long ago. I recall being frustrated after beating my head against it for a while and making no headway.

Filled with renewed vigor, I tackled it again here under the invigorating Mexican sunshine. Alas, the sunshine proved to be of little help. I spent a long time wrangling with the equation in the form y² = (x+1)(x-1)/D. This led me to explore the family of solutions nD + 1 and nD – 1, which are very nice, but definitely not minimal.

I then played around with factoring D and trying to come up with some way to efficiently search for potential solutions using those factors, since D has to divide evenly into (x + 1)(x – 1). But it needn’t do so in any convenient way. For example, for D = 28, the minimal solution is (127 + 1)(127 – 1)/28 which factors into 2 · 2 · 2 · 2 · 2 · 2 · 2 × 2 · 3 · 3 · 7 / 7 · 2 · 2. Some of the factors go into the left numerator and some in the right, it’s mayhem.

Eventually I decided to do some Googling and discovered that this is closely related to the well-known Pell equation. For reasons not entirely clear to me, its solutions lie in the continued fraction expansion of the square root of D.

Implementing the “usual recurrence relations” was pretty straightforward, though a bit of memoization was needed (the unsightliness of that part of the code annoys me, but I am annoyed enough with this problem already that I don’t feel like twiddling it further). BigInt also turned out to be necessary, as the largest “minimal” x is 16421658242965910275055840472270471049, which doesn’t quite fit into 64 bits.

Euler 65

2011-01-13T00:00:00-08:00

Problem 065: (source):

object Euler065 extends EulerApp {
  case class Frac (numer :BigInt, denom :BigInt) {
    def + (n :BigInt) = Frac(n * denom + numer, denom)
    def invert = Frac(denom, numer)
  }
  def compute (count :Int, n :Int) :Frac = {
    val term = if (n == 1) 2 else if (n % 3 == 0) 2*(n/3) else 1
    if (n == count) Frac(term, 1)
    else compute(count, n+1).invert + term
  }
  def answer = compute(100, 1).numer.toString map(_-'0') sum
}

The only challenge here is algorithmizing the computation of the continued fraction, while preserving the numerator and denominator, so that we can compute the answer from the final numerator. I think the case class makes the calculations on the fraction much tidier.

The eternal quest for personal information management nirvanaz

2010-11-28T00:00:00-08:00

I have progressed through a broad variety of personal information management tools over the course of my life, and they invariably fail to meet my needs in one way or another. This periodically motivates a flurry of note taking on what would comprise the “perfect” personal information management system, which I generally record using whatever duct tape and baling wire contraption I am currently using to keep track of things, and then promptly forget about.

About half a decade ago, I wrote a standalone application that handled the parts of PIM that were done wrongly, in my opinion, by other tools (keeping lists of tasks to do, a journal of daily doings, and seeing at a glance what events were upcoming and recently past). It kept the urges to write a more comprehensive system at bay, and I got by with my tool, a haphazard collection of wikis and text files, and Google calendar.

My entry into the world of computer science research last year triggered a dramatic increase in my generation of notes and ideas that needed keeping track of, not to mention a much more finely divided schedule. To meet my increased PIM needs, I tried minimalist solutions, maximalist solutions, and Googly solutions (among a dozen others ...). Each rewarded my substantial organizational efforts with eventual dissatisfaction and malaise. My resolve eventually eroded and I succumbed to my perennial urge to write my own all-singing, all-dancing, wiki/todo/calendar/datastore/dwim application.

I thus present Spare Cortex, an unabashed indulgence in my own personal information management needs. To atone for the sin of polluting the world with yet another wiki, I have strived to make it usable by people other than its author. I have provided a demonstration of its basic functionality to help interested parties determine whether it is worth the time investment of trying out.

My motivations in creating the system (which you may compare to your own PIM motivations to gauge your interest) were as follows:

It should not extravagantly waste a quarter or more of its screen real estate on headers, banners, navigation links and other wikijunk (to paraphrase Tufte).
It should be hierarchical. The flat namespace of most wiki systems breaks down once you add half a dozen projects each with a dozen pages of content. Prefixing the name of every page with the name of its parents scales poorly.
It should support structured content (most importantly checklists, and what I call “pages”) natively. Configuring half a dozen poorly implemented and unmaintained plugins is a punishment I’d rather avoid.
With regard to checklists, having to open the wiki editor and scroll down through a sea of text to find a todo item to check off (or edit, or move up or down in priority) is painful.
With regard to pages, any project that does not die an early death will eventually accumulate sort of “dashboard” at its top-level, containing outstanding todo items, lists of useful resources, and the top-level links to the copious notes that you’ve taken as the project evolves. Basic page layout capabilities enable one to organize this dashboard page in a way that allows easy viewing of what’s going on with the project and easy updating of ephemeral data.
Furthermore, I don’t want that data landlocked amidst semi-structured text. I should be able to see all of the tasks across all of my projects, or tag individual tasks and place a view of tasks with e.g. the today tag on my top-level “life dashboard” page, alongside my appointments and journal (note: the tagging feature is not yet complete, but is soon forthcoming).
Parts of pages, like checklists, should be accessible at individual URLs which are mobile web browser friendly, so that you can readily access any task list while on the go, and quickly add whatever thought popped into your head.
It should support the attachment of media to individual pages, again not constrained by a global namespace.
It should be searchable (not yet implemented) and support bulk import and export (also not yet implemented).
It should support access controls to enable sharing subtrees (or an entire tree) in a read-only or read-write manner with other people, or with the anonymous public (partly implemented).

The system is implemented on Google’s AppEngine, which I confess to having mixed feelings about. Its reliability has been less than stellar during the course of development, though it seems to be improving lately. Further rants on that subject should occupy their own blog post. Should you make use of Spare Cortex for your own information management needs, you can rest comfortably in the knowledge that I’m highly motivated to keep it running smoothly, owing to my own repeated daily use.

If you’ve managed to read this far and are not simply in shock at this display of raw enthusiasm for something of such little consequence, please give Spare Cortex a whirl.

WTFF?

2010-09-14T00:00:00-07:00

I saw some weird background activity on my network interface and undertook an investigation. netstat readily revealed that Firefox had a metric ton of TCP connections open:

yonami.local:58809      cdce.sef004.interna:www ESTABLISHED 7215/firefox-bin
yonami.local:58828      cdce.sef004.interna:www ESTABLISHED 7215/firefox-bin
yonami.local:33425      pv-in-f106.1e100.ne:www ESTABLISHED 7215/firefox-bin
yonami.local:58829      cdce.sef004.interna:www ESTABLISHED 7215/firefox-bin
yonami.local:46025      208.50.77.78:www        ESTABLISHED 7215/firefox-bin
yonami.local:46001      208.50.77.78:www        ESTABLISHED 7215/firefox-bin
yonami.local:41860      208.50.77.95:www        ESTABLISHED 7215/firefox-bin
yonami.local:45853      pv-in-f100.1e100.ne:www ESTABLISHED 7215/firefox-bin
yonami.local:45339      px-in-f102.1e100.ne:www ESTABLISHED 7215/firefox-bin
yonami.local:58807      cdce.sef004.interna:www ESTABLISHED 7215/firefox-bin
yonami.local:58801      cdce.sef004.interna:www ESTABLISHED 7215/firefox-bin
yonami.local:41861      208.50.77.95:www        ESTABLISHED 7215/firefox-bin
yonami.local:41864      208.50.77.95:www        ESTABLISHED 7215/firefox-bin
yonami.local:41862      208.50.77.95:www        ESTABLISHED 7215/firefox-bin
yonami.local:41863      208.50.77.95:www        ESTABLISHED 7215/firefox-bin
yonami.local:58808      cdce.sef004.interna:www ESTABLISHED 7215/firefox-bin
yonami.local:56984      185-131.amazon.com:www  ESTABLISHED 7215/firefox-bin
yonami.local:52057      208.50.77.72:www        ESTABLISHED 7215/firefox-bin
yonami.local:41865      208.50.77.95:www        ESTABLISHED 7215/firefox-bin

This was especially peculiar, because the only Firefox window I had open was to some totally static webpage. No Comet or other JavaScript funny-business going on there. No other Firefox tabs or windows open. So what the hell are all these other connections?

Googling reveals that 1e100.net are Google servers and that Firefox may have connections open to these to do Safebrowsing or other random bits. The others are mysterious: cdce.sef004.internap.net is probably some Internap CDN service, the 208.50.77.xx addresses all live behind verio-1.ar5.SEA1.gblx.net, and look like Akami servers. 185-131.amazon.com doesn’t even forward resolve (I didn’t save a copy of the netstat -na output).

Nothing looks especially nefarious, but why would I have so many lingering established connections? Surely if I close a tab, Firefox closes all keep-alive connections originating from that tab. Of course, when I killed Firefox, the network activity stopped.

Meanwhile, Chrome was running with tabs open to Gmail, another Gmail and Google Calendar. It’s open connections were explicable and substantially fewer:

yonami.local:45460      pv-in-f83.1e100.net:www ESTABLISHED 3002/google-chrome
yonami.local:43754      pv-in-f139.1e100.:https ESTABLISHED 3002/google-chrome
yonami.local:49226      pv-in-f83.1e100.n:https ESTABLISHED 3002/google-chrome
yonami.local:35690      pv-in-f17.1e100.n:https ESTABLISHED 3002/google-chrome
yonami.local:40323      pv-in-f83.1e100.net:www ESTABLISHED 3002/google-chrome

I won’t even start on Firefox’s memory utilization “patterns”. Either way, score another point for Chrome’s process-per-tab resource management model.

Better symlink behavior for Git and Mercurial

2010-09-11T00:00:00-07:00

Like most developers, you probably don’t like unnecessary typing. By that, I don’t mean the static and dynamic variety, but the fingers on keyboard variety. However, when it comes to manipulating files via version control, we have recently taken a step backward.

Historically, when faced with paths that look like the following:

src/main/java/com/company/project/foopkg/MyAwesomeFoo.java
src/main/java/com/company/project/barpkg/MyAwesomeBar.java
src/main/java/com/company/project/bazpkg/MyAwesomeBaz.java
src/test/java/com/company/project/foopkg/MyAwesomeFooTest.java

I have created symlinks at the top-level of my project as follows:

% ln -s src/main/java/com/company/project code
% ln -s src/test/java/com/company/project tcode

This allows me to do things like:

% svn diff code/foopkg/MyAwesomeFoo.java

which is reasonably concise and fairly tab-completion-friendly. Subversion happily expands the symlink and just does the right thing. Great.

Enter Git and Mercurial. For not entirely explicable reasons (save yourself the pain of Googling “git symlinks” or “hg symlinks” and perusing the results; it’s a wasteland of madness), they behave as follows in such circumstances:

% git diff code/foopkg/MyAwesomeFoo.java
(tumbleweeds)
% hg diff code/foopkg/MyAwesomeFoo.java
abort: path 'code/foopkg/MyAwesomeFoo.java' traverses symbolic link 'code'

Git is especially disingenuous in simply reporting that there are no diffs. At least if you try to do something like commit a file that’s on the other end of a symlink it complains:

% git commit code/foopkg/MyAwesomeFoo.java
error: pathspec 'code/foopkg/MyAwesomeFoo.java' did not match any file(s) known to git.

However, what we really want these tools to do is DWIM: just figure out where the damned file is and do what I asked.

Fortunately, both tools behave themselves when presented with fully qualified paths. I whipped up a little Perl script to hide the scary symlinks from these poor overburdened tools. I’ve reproduced it here, in case you too, dear reader, find this to be an annoyance:

#!/usr/bin/perl -w
#
# Expands symlinks before Git/Hg sees them.

use Cwd 'abs_path';

# Will the real DVCS please stand up?
my ($x,$orig) = ($0 =~ m#^(.*/)?([^/]+)$#);
my $bin;
foreach (`which -a $orig`) {
    chomp;
    $bin = $_ unless ($_ eq $0);
}
die "Can't find $orig binary.\n" unless $bin;

# Expand any symlinks in our args
my @args = ($bin);
foreach (@ARGV) {
    my ($dir, $name) = ($_ =~ m#^(.*)/([^/]+)$#);
    if ($dir && -d $dir) {
        my $abspath = abs_path($dir);
        if ($abspath !~ m/$dir$/) {
            push @args, "$abspath/$name";
            next;
        }
    }
    push @args, $_;
}

# Now pass the processed arglist on to the real deal
exec @args;

You can stuff this into files named git and hg in your path (and chmod a+rx them), or you can use more drive space and stick it in something like dwim-dvcs and symlink git and hg to the single file.

If you are conversant in the line-noise that is Perl, you may notice that I go to the trouble of handling files that don’t exist. That way if you do something like the following, it still works:

% git rm code/fookpkg/MyAwesomeFoo.java
% git commit code/fookpkg/MyAwesomeFoo.java (file no longer exists at this point)

It’s still possible to confuse the script (in which case it just passes your arguments through as-is to git/hg), but at least you have to go further out of your way to cut yourself on that edge case.

Why Maven Sucks: Act I, Reprise

2010-09-10T00:00:00-07:00

The XML insanity of passing compiler arguments to Maven like so:

<compilerArguments>
  <Xlint/>
  <Xlint:-serial/>
  <Xlint:-path/>
</compilerArguments>

has not surprisingly, come back to bite me in the ass. Most normal Maven actions seem to have no problem with it, but when I try to actually publish my project (via mvn release:prepare), I get the error I expected to see all along:

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error reading POM: Error on line 104: The prefix "Xlint" for element "Xlint:-serial" is not bound.

This inspired further investigation into this problem. Surely among the thousands of projects built with Maven, someone else is trying to pass one of javac’s myriad options that include a colon (e.g. -proc:only, -Xlint:-foo, -g:lines, -Xprefer:source, -Xbootclasspath/p:..., etc.).

I eventually found this improvement request, which has remained unresolved for three years! Is passing arguments to the compiler not one of the most fundamental things a build system has to do? Even make gets that right.

Someone even submitted a patch in November ‘09 to make <compilerArgument> work as advertised. It has not been applied. The issue has not been assigned to anyone. As far as I can tell the Maven developers are not even aware that this is a problem. I’m not even sure these “Maven developers” actually exist. More likely, Skynet sent Maven down to slowly sap the will of the programmers that created it, ensuring that they never render it obsolete by releasing a newer version.

Update: I’ve since discovered that this seems to be the practiced workaround:

<fork>true</fork>
<compilerArgument>-Xlint" "-Xlint:-serial" "-Xlint:-path</compilerArgument>

Yes. Really.

Better symlink behavior for Git and Mercurial

2010-09-10T00:00:00-07:00

Historically, when faced with paths that look like the following:

src/main/java/com/company/project/foopkg/MyAwesomeFoo.java
src/main/java/com/company/project/barpkg/MyAwesomeBar.java
src/main/java/com/company/project/bazpkg/MyAwesomeBaz.java
src/test/java/com/company/project/foopkg/MyAwesomeFooTest.java

I have created symlinks at the top-level of my project as follows:

% ln -s src/main/java/com/company/project code
% ln -s src/test/java/com/company/project tcode

This allows me to do things like:

% svn diff code/foopkg/MyAwesomeFoo.java

which is reasonably concise and fairly tab-completion-friendly. Subversion happily expands the symlink and just does the right thing. Great.

% git diff code/foopkg/MyAwesomeFoo.java
(tumbleweeds)
% hg diff code/foopkg/MyAwesomeFoo.java
abort: path 'code/foopkg/MyAwesomeFoo.java' traverses symbolic link 'code'

Git is especially disingenuous in simply reporting that there are no diffs. At least if you try to do something like commit a file that’s on the other end of a symlink it complains:

% git commit code/foopkg/MyAwesomeFoo.java
error: pathspec 'code/foopkg/MyAwesomeFoo.java' did not match any file(s) known to git.

However, what we really want these tools to do is DWIM: just figure out where the damned file is and do what I asked.

#!/usr/bin/perl -w
#
# Expands symlinks before Git/Hg sees them.

use Cwd 'abs_path';

# Will the real DVCS please stand up?
my ($x,$orig) = ($0 =~ m#^(.*/)?([^/]+)$#);
my $bin;
foreach (`which -a $orig`) {
    chomp;
    $bin = $_ unless ($_ eq $0);
}
die "Can't find $orig binary.\n" unless $bin;

# Expand any symlinks in our args
my @args = ($bin);
foreach (@ARGV) {
    my ($dir, $name) = ($_ =~ m#^(.*)/([^/]+)$#);
    if ($dir && -d $dir) {
        my $abspath = abs_path($dir);
        if ($abspath !~ m/$dir$/) {
            push @args, "$abspath/$name";
            next;
        }
    }
    push @args, $_;
}

# Now pass the processed arglist on to the real deal
exec @args;

You can stuff this into files named git and hg in your path (and chmod a+rx them), or you can use more drive space and stick it in something like dwim-dvcs and symlink git and hg to the single file.

If you are conversant in the line-noise that is Perl, you may notice that I go to the trouble of handling files that don’t exist. That way if you do something like the following, it still works:

% git rm code/fookpkg/MyAwesomeFoo.java
% git commit code/fookpkg/MyAwesomeFoo.java (file no longer exists at this point)

It’s still possible to confuse the script (in which case it just passes your arguments through as-is to git/hg), but at least you have to go further out of your way to cut yourself on that edge case.

Why Maven Sucks: Halftime Show

2010-09-09T00:00:00-07:00

One can, and is indeed encouraged to, sign one’s Maven artifacts with GPG. This involves generating an artifact-version.jar.asc file containing the ASCII representation of the cryptographic signature for the artifact in question. This is uploaded to the Maven repository along with the artifact.

Maven then helpfully generates artifact-version.jar.asc.md5 and artifact-version.jar.asc.sha1, and uploads those to the repository as well. I don’t need one, let alone two, cryptographic hashes to tell me that my cryptographic signature has not been tampered with. This is no doubt because Maven generates cryptographic hashes for everything that goes into the repository, but the result seems somewhat comical.

I tried to track down the reason for generating hashes with two different algorithms, but the commit that introduces the change is cryptic:

[MINSTALL-34] update dependencies and correct build.

I can only assume that this was a design by committee decision of some sort.

While I was perusing the commit logs, I saw that someone else shared my opinion about the utility of hashing signatures:

r630088 | wsmoak | 2008-02-21 20:27:13 -0800 (Thu, 21 Feb 2008) | 3 lines

[MINSTALL-48] Don't create checksums for gpg signature files
Submitted by: Niall Pemberton

I can only assume that my maven-install-plugin is defaulting to some version that’s more than two and a half years old because I haven’t gone out of my way to add five lines of boilerplate to my POM to ensure that I’m using the latest version of a plugin that I had never heard of until I undertook this investigation.

Why Maven Sucks: Act III

2010-09-08T00:00:00-07:00

The saga (act i, act ii) continues.

More reasons why Maven sucks:

Plugin version handling sucks.

I learned this when I was trying to get Maven to generate javadocs. I was configuring the Maven Javadoc plugin with links to external javadocs when I discovered the very handy detectJavaApiLink option. It automatically adds a link to the standard Java API javadocs whose version matches the version specified in the -source argument to your compilation.

Imagine my chagrin when I discovered that it was not working. I tried enabling it manually, even though it claims to be enabled by default. I then ran Maven with debugging output to see if it would report what options it was passing to the javadoc tool, and happened to notice that detectJavaApiLink didn’t appear at all in the dump of the plugin configuration.

This inspired me to investigated the documentation and see that this option was added in version 2.6 of the plugin. I inspected my debugging output and saw that I had version 2.5. I had already chafed against the fact that all of the documentation I saw for Maven on the web contained POM snippets with arbitrary and almost universally out of date versions for the plugins. My solution to this was to simply remove the <version> element entirely from my plugin declarations.

I assumed, foolishly, that this would cause Maven to use the latest version of the plugins. In fact, it seems to use some arbitrary version. My POM inherits from a POM which actually references version 2.7 of the javadoc plugin, so I have no idea where it’s getting version 2.5. Perhaps it is some transitive dependency of some other plugin whose version number I have not specified.

Further research into this matter revealed that I can specify RELEASE as a version and get the latest release version. Peachy. However, this is considered bad form by the Maven community because it may cause problems with your build if plugins decide to change the default values of their options. Is this really such a big problem that it’s better to have everyone expend the effort to periodically manually upgrade their dozens of Maven plugins in their dozens of builds? Furthermore, is it such a big problem that it’s worth having tens of thousands of web pages filled with instructions on how to do X or Y with Maven, all filled with ancient version numbers just waiting to bite newbie/lazy/tired Maven users in the ass?

It’s additionally annoying that all of the plugins implicitly included in your build are of some arbitrary version. If you want the latest version of those, you have to add to your already dangerously obese POM file an additional 5 lines of boilerplate for each plugin you wish to keep up to date.
Maven could have magically handled Javadoc dependencies, but didn’t.

I also noticed, while poring over the documentation for the javadoc plugin, an option entitled detectLinks, which purports to automatically add -link arguments for the projects on which your project depends. Awesome! A longstanding annoyance with javadoc is that you have to manually configure it with URLs for every project whose types you reference in your own project so that it correctly links those types in the generated documentation.

Maven goes to all this trouble to collect project metadata and publish it along with the project’s artifacts. They must have included the project’s javadoc API url among that metadata, and that is what detectLinks is using to automatically wire up the documentation.

Nope. They did nothing nearly so useful. One of Maven’s dead-on-arrival ideas from the early days was that it could generate a cookie-cutter website for your project, full of a bunch of boilerplate crap. In generating such websites, it would put your API documentation at ${project.url}/apidocs, and that’s exactly where the javadoc plugin expects to find the API docs for all of your dependent projects.

You can take a guess at how many of my dependent projects have their API documentation at that URL. I’ll give you a hint, you can count the number on zero hands.

Why Maven Sucks: Act II

2010-09-08T00:00:00-07:00

Sun, in their infinite wisdom, circa Java 1.3, extended the jar file specification to allow dependencies to be expressed inside jar files by adding a Class-Path attribute to the MANIFEST.MF file. This causes the JVM (and javac, and various app servers) to try to magically add dependent jar files to the classpath. This is half-assed and wrong in too many ways to enumerate here.

Some other enthusiastic Sun engineer then helpfully added activation.jar to mail.jar’s Class-Path attribute, under the multiple misguided assumptions that no one would ever possibly need to use mail.jar without also having activation.jar in their classpath, with precisely that name, and located in precisely the same directory. Great.

Maven then upped the ante on this little fiasco by deciding that any time the Java compiler generates a warning that they can’t parse, they should fail the build. Clearly it’s critical that your build system be conversant in every possible warning that might be emitted by your compiler. As a result, when I fix their boneheaded decision to suppress warnings by default, my build now fails with this demonstration of awesomeness:

could not parse error message: warning: [path] bad path element "/home/mdb/.m2/repository/javax/mail/mail/1.4.1/activation.jar": no such file or directory

Thank you Sun, and thank you Maven.

For those of you arriving from the Googles and looking for a solution to this problem, it is as follows: pass -Xlint:-path to javac to prevent it from issuing a warning when it cannot find activation.jar. If you will be passing this compiler argument via Maven, be sure to read Why Maven Sucks: Act I before doing so, to avoid the various pitfalls that await.

I should further mention the irony that activation.jar is listed as a dependency of mail.jar, but it’s being placed in .m2/repository/javax/activation/activation/1.1/activation-1.1.jar which is where Maven wants it and not where the utility deficient Class-Path mechanism expects it to be.

Why Maven Sucks: Act I

2010-09-08T00:00:00-07:00

These ruminations were languishing in commit messages, which would normally satisfy my urge to complain. But now I need to complain further, and don’t have a commit message in which to express my discontent, so I’m airing the whole basket of dirty laundry here, and going on the record as a Maven hater. For those of you privy to the original commit messages, my new complaints will be aired in Act III. Now then, on with the show.

Reasons that Maven sucks:

Warnings and deprecations are not shown by default.
Documentation shows use of <compilerArgument> with multiple arguments in a single element:

<compilerArgument>-foo -bar</compilerArgument>

which is a bald-faced lie. Only a single argument is allowed inside a <compilerArgument> element. If you try to use multiple arguments, you see that it uselessly tries to pass everything as a single argument:
```
 Failure executing javac, but could not parse the error:
 javac: invalid flag: -foo -bar
 Usage: javac <options> <source files>
 use -help for a list of possible options
```
(Bear in mind that -foo and -bar were actual javac arguments, not the stand-ins I’m using here. I’m saving the real arguments for a punchline below.)

Web search turns up “helpful” advice to use multiple elements:
```
 <compilerArgument>-foo</compilerArgument>
 <compilerArgument>-bar</compilerArgument>
```
Fair enough, but also a bald-faced lie. After spending a bunch of time debugging why my compiler arguments were not working, I discovered that Maven was just (silently) using the last one and ignoring/overwriting all of the previous arguments.

I had noticed while perusing the documentation that it was also possible to use the so-called “Map version” (whatever that means), which uses this completely stupid syntax:
```
 <compilerArguments>
   <foo/>
   <bar/>
 </compilerArguments>
```
Why is that syntax completely stupid, you might ask? Well, dear reader, because the arguments that I’m actually passing are the following:
```
 <compilerArguments>
   <Xlint/>
   <Xlint:-serial/>
 </compilerArguments>
```
which is a case study in how not to represent information in XML. I didn’t even try that originally because I was sure that it would not work, given the wacky non-[a-zA-z]+ nature of the argument I needed to supply. The fact that it does work gives me the fear. You might think the same thing Charlie did, when he replied to my original commit message:

> Doesn’t that create an element named -serial in the Xlint namespace?

Apparently not, because it works.

You might also wonder, like I did, whether the following form would provide satisfaction:
```
 <compilerArguments>
   <compilerArgument>-Xlint</compilerArgument>
   <compilerArgument>-Xlint:-serial</compilerArgument>
 </compilerArguments>
```
By being absurdly verbose, it seems right in line with The Maven Way™. However, that results in -compilerArgument=-Xlint and -compilerArgument=-Xlint:-serial being passed to the compiler. Hilarity naturally ensues.

Shoot me now

2010-08-25T00:00:00-07:00

I’m foolishly trying to write a javac annotation processor in Scala. This has already been a road fraught with gratuitous obstacles, but the latest wrinkle has driven me to vent in public.

When I run my annotation processor thusly:

javac -processor FooProcessor -processorpath fooproc/classes:blah/scala-library.jar SomeFile.java

I get java.lang.IllegalStateException: zip file closed when my processor tries to load Scala classes. But if I unpack scala-library.jar into fooproc/classes then it works. Joy! Somehow the scala-library.jar file causes javac to choke.

I of course tried repacking the jar file, in case perhaps something in the stock scala-library.jar was weird, but using the repacked version still failed with zip file closed. What’s even more awesome is that javac 1.6 does not have a problem with scala-library.jar, only javac 1.7.

Euler 2⁶

2010-08-25T00:00:00-07:00

Problem 064: (source):

object Euler064 extends EulerApp {
  case class Root (root :Int, add :Int, div :Int) {
    def expand = {
      val term = ((math.sqrt(root) + add)/div).toInt
      val nadd = term*div - add
      (term, Root(root, nadd, (root - nadd*nadd)/div))
    }
    override def toString = "(â" + root + "+" + add + ")/" + div
  }
  def expansion (terms :List[Int], roots :List[Root]) :List[Int] = {
    val (term, root) = roots.head.expand
    if (root.div == 0 || roots.contains(root)) term :: terms
    else expansion(term :: terms, root :: roots)
  }
  def answer = 1 to 10000 map(n => expansion(Nil, Root(n, 0, 1)::Nil).length) count(_%2==0)
}

Here we have a nice little case class to represent a single step in the infinite expansion. The process of generating the next expansion could be done purely with simple arithmetic, but I’m lazy, so I just use math.sqrt to obtain the non-fractional part. I don’t actually need to keep track of the terms to obtain the solution, but it was handy to have when I was validating the solution, so I kept it around. Same goes for toString. I just love that little root sign!

Euler 63

2010-08-21T00:00:00-07:00

Problem 063: (source):

object Euler063 extends EulerApp {
  def pows (n :Int) = Stream.from(1) prefixLength(p => BigInt(n).pow(p).toString.length == p)
  def answer = 1 to 9 map(pows) sum
}

The main observation here is that the number of digits of a^x, for a ≥ 10, is guaranteed to exceed x. So we can restrict ourselves to looking only at the numbers from 1 to 9. Furthermore, the number of digits of a^x, for 1 ≤ a ≤ 9, will equal x up to some maximum x, and then be less than x for all higher x. That enough xs for you?