I can't speak with much authority on your Apache memory spikes, but I can offer a theory:

Apache uses two models: process per request and thread per request. These models are very similar and share a fundamental flaw (versus the async approach used by Nginx and Lighttpd): each request consumes a relatively large amount of memory *for the duration of the request*. This might sound like a "duh" prospect (and proponents of the async method will certainly agree), but you have to consider what this means in the context of slow clients or large downloads: if you have 1000 simultaneous requests, there will be 1000 threads/processes alive handling them. If these requests are handled quickly then it's not much of an issue (and this is typical for a webserver). However, if the client connections are slow, or Apache is proxying to a backend that is slow (frameworks such as Rails, TurboGears, Pylons, Wordpress can only handle around 30 req/s on typical hardware), then these processes will be alive much longer than expected. It's sort of like an unintentional DoS attack, consuming memory and resources such as TCP sockets.
In a VPS situation, these resources are already quite limited compared to a dedicated server, so what you're probably seeing is this:

1. lots of requests come in, Apache starts consuming memory
2. because the VPS has limited memory, swapping starts earlier than expected
3. swapping causes further slowdown in the VPS and in handling client connections, causing more processes to exist simultaneously, hence using more memory and further exacerbating the problem.

This is all speculation, but I think you'll agree it's a quite likely scenario.

Regards,
Cliff

Just as an FYI, here's a couple quotes from Nginx users running moderately large sites. In fact the quote from Bob Ippolito was from a conversation on the TurboGears list that convinced me to try Nginx in the first place (I was previously using Pound to proxy to Apache and Lighttpd):

http://wiki.codemongers.com/NginxWhyUseIt

I don't have first-hand knowledge, but I believe Bob's site is running on a single, fairly modest server, if that matters.

Also, the note on the wiki about Nginx "running on many heavily loaded Russian sites" is something of an understatement: Nginx powers the largest hosting company in Russia (and I believe it was originally commissioned by them). It's a bit like saying that AOLserver powers some heavily loaded US sites ;-) The fact that it's barely known outside of Russia has been (until recently) mostly due to the fact that the documentation was all in Russian.

Of course, the fact that I was able to replace Apache, Lighty and Pound for around 60 customers in one night without any English documentation (relying on only a few examples I found in blogs) speaks volumes about how sane Nginx' configuration is. Imagine having almost no Apache documentation and doing the reverse ;-) Sometimes simpler is better. If you need esoteric modules (e.g. mod_svn), then there's probably nothing to match Apache. If you just need to serve up a typical website (i.e. 80% of what's out there), then Nginx wins hands down on almost every count.

Regards,
Cliff