Interesting Stuff

Warning: this post got embarrassingly long. For the short version, read up through the paragraph that starts with “If you use.”

This is something that’s been bugging me for a while, and I’d like to lay it to rest. If you’ve done any work in C++, you’ve probably heard the riddle about how to swap two integers without using a temporary variable. The question is exactly what it sounds like: you have two integers (call them x and y), you want to swap their values, and you may or may not want to use a third, temporary variable while doing it. Here are some ways of swapping integers that I have actually seen in professional code written by professional coders:

{  // Limit the scope of t
  int t = x;
  x = y;
  y = t;
}

x ^= y;
y ^= x;
x ^= y;

x ^= y ^= x ^= y;

x = x + y;
y = x - y;
x = x - y;

std::swap(x, y);

Try to come up with some answers before reading further.

When you’ve decided on your answers, let’s take a closer look. No, on second thought, this got really long. For those of you who don’t want to read all the way to the end, let me just get my main point out of the way. After that, we’ll take a closer look. If you don’t want to spoil the ending, skip the next paragraph.

If you use this link you will spoil the ending, so skip to the next paragraph after this one. →

Suppose you could create the specification for your ideal programming language. It must be possible to implement (it can’t solve undecidable problems, etc), but other than that the sky’s the limit. What would you put in it (what kind of syntax, intrinsics, etc)? For what sort of applications would it be used? What other features would it have? Would the interpreter/compiler do anything unusual? Tell me anything you want about your ideal language and the tools that go with it. You’re welcome to answer some of these questions if you don’t have answers to all of them.

I’ve been asking variations of this question to all sorts of people, and it’s fascinating seeing where they agree and where they don’t.

So, even before I get to the what the algorithm does (which is quite interesting on its own, even before you find out how it accomplishes that), the very first intriguing part is its name. It’s officially referred to as the Blum-Floyd-Pratt-Rivest-Tarjan Algorithm. Is your mouth watering yet? I thought so.¹ However, CLRS refers to it as the Select Algorithm (see section 9.3), so that’s what I’m going to call it. If you have an unsorted array of n items, it’s a worst-case O(n) algorithm for finding the i^th largest item (often the median, since that’s a commonly used statistic). Here’s how the algorithm works:

1. Put all the elements of the input into groups of 5 (with one group at the very end that could have less than 5 elements, of course). Find the median of each group.
2. Recursively use the Select algorithm to find the median of all the group-of-5 medians from step 1. Call this median of medians M.
3. Take all the items from the input and partition them into the group of items less than M and the group of items bigger than M.
4. If M is the i^th biggest, that’s the answer, and we’re done. If it’s not, you know which partition contains the i^th biggest item, and you can recursively call Select on that group (with a different i if the group it’s in is the group of items smaller than M).

To give an example of that last statement, suppose we’re looking for the 65th biggest number from a group of 100 numbers, and we find that M is the 53rd largest. There are 47 numbers smaller than M, and when we recurse, we’ll now be looking for the 12th largest element of that set of 47. If instead M had been the 68th largest number, in our recursive step we’d look at the 67 numbers larger than M and continue to look for the 65th largest one.

Now, I claimed the running time is linear, and I’d better back up that claim. The trick is to consider how many items get thrown out in that last step. Half the group-of-5 medians are larger than M, and half are smaller. Without loss of generality, suppose the i^th largest item is in the group that is larger than M. Now, not only do we know that we can henceforth ignore the half of the group-of-5 medians smaller than M, for each of those medians we can ignore the two items in the group of 5 that were smaller than that median. In other words, we can eliminate at least 3/10 of the elements we were searching over (we can eleminate 3/5 of the items in the groups whose median is smaller than M, and half of all the groups fit this category). Sure, I’m eliding an additive constant for the items in the group with M itself (not to mention the items in that last small group at the end when n is not a multiple of 5), but that’s not going to affect the asymptotic running time.

So now we can build the recurrence relation at work here. If T(n) is the amount of work done to perform Select on n items, we have

T(n) = O(n) + T(n/5) + T(7n/10)

That is to say, we do O(n) work in steps 1 and 3, we do T(n/5) work in step 2, and we do T(7n/10) work in step 4 because we know at least 3/10 of the elements are in the wrong partition. I’ll leave it as an exercise to the reader to draw out the recursion tree and find the actual amount of work done, but I assure you T(n) is in O(n).

The crazy part here is that magic number 5: If you do this with 3 items per group, the running time is O(n log n). If you do it with more than 5, the running time is still linear, but slower (you spend more time in step 1 finding all the initial medians). The optimal number of items per group really is 5. Nifty!

[1] For those of you not in the know, these are all really famous algorithms names. Blum is the co-inventor of the CAPTCHA. Floyd is from the Floyd-Warshall all-pairs shortest-paths algorithm, which is a staple in introductory algorithms courses. Pratt is from the Knuth-Morris-Pratt string matching algorithm, which was groundbreaking when it was new but is nearly obsolete now. Rivest is the R in RSA encryption, one of the most widely used encryption schemes around. Tarjan is the co-creator of splay trees, a kind of self-balancing tree with the handy property that commonly accessed items float to the top. Blum, Floyd, Rivest, and Tarjan have each won the Turing Award, which is like the Nobel Prize of computer science.

I’ve been working off and on to find the optimal strategy for the final round of a game of liar’s poker. I’ve gotten a bit farther on it, and it now looks solveable. I’ve now solved the 12-card deck version (i.e. if the deck only contains aces, kings, and queens), and I suspect I’ve got an algorithm that will solve the whole problem in a reasonable amount of time.

A few more lemmata →

I recently wrote about a card game called Liar’s Poker and my endeavours to find an optimal strategy for the final round. Some extra progress has been made, but it’s still a long ways off. Despite what I wrote when I edited my previous post, I’m no longer convinced that we can adapt Simplex or a gradient descent algorithm to solve this. It turns out that although the gradient function is piecewise linear, there are an exponential number of different linear regions in it, and I’m not convinced we can do something like Simplex without enumerating them all. but here’s some other stuff Reid and I have done on the problem:

More strategies and proofs and stuff →

Liar’s Poker is a card game we sometimes play around the office. It’s a pretty simple game to learn if you’re already familiar with poker hands: the rules to Liar's Poker, followed by some work figuring out the strategy for the final round →

This year, the Algorithms course at Mudd is using a new book, written by Dasgupta, Papadimitriou (the complexity theorist), and Vazirani (the quantum computing guy). You can view a July 2006 draft of the book, which actually looks pretty nice. It doesn’t cover all that I think an algorithms course should (in particular it handles the standard data structures cursorily at best), but it definitely skips the boring bits and goes right to the fun parts. It starts out with cryptography, then goes into matrix multiplication and the FFT (which sound boring, but are actually pretty neat problems). It even has a chapter on quantum computing!

I don’t think this will ever replace CLRS in terms of completeness, but it’s a lot more fun. Definitely worth a read if you’re looking for an algorithms textbook. and really, who isn’t looking for one these days?

(found while reading the current XKCD)

Java’s “random” number generator has some surprisingly simple patterns in it. Nearly all random number generators on computers are pseudo-random, meaning that they only seem random-ish and are not truly random (with some notable but impractical exceptions). This happens because computers themselves are deterministic and therefore inherently non-random. Most pseudo-random number generators these days use the previous “random” value as an input to create the new value, which will be used to create the next one after that (which is why plots like these are often used to evaluate the effectiveness of a PRNG: by plotting the current value with the previous one, you can see how much the values get scattered around the range of outputs). As an aside, I should note that plotting the same thing in 3-D (with the current value, the previous value, and the value before that as the 3 coordinates) can reveal other problems, such as the problems in Randu, an early PRNG. Anyway, because most PRNGs use their previous value to compute the next one, they will cycle after they get the same value twice. A good PRNG that generates n-bit numbers shouldn’t cycle until it has generated 2ⁿ of them. Java’s implementation appears to be much worse than this, which is disappointing. Come on, guys, you can do better! The cryptographically secure hash is my favourite, and I’m currently trying to prove that NC≠P using a variation of this. If it’s any consolation to Sun (the creators of Java), Matlab has similar problems.

Define a “hub” to be a vertex that shares an edge with every other vertex (such as the middle of a star graph, or any vertex in a complete graph). Suppose we have the adjacency matrix of an undirected, unweighted graph with V vertices (so our input has size V^2). Find an algorithm with running time o(V^2) that can determine whether a hub exists in the graph, or prove that no such algorithm exists. Note the use of little-o notation: the algorithm must be asymptotically faster than (big) O(V^2).

My bet is that no such algorithm exists, but I can’t figure out how to prove it.

I’ll be at Mudd tomorrow, btw.