A little known fact is that the non-server version of Mac OS X ships with a VPN server, but you’ll have to go down to the command line to configure it. There’s various bits of information about the web on how to do this, so I thought I’d wrap it all up here.

1. An article on macosxhints.com will get you started. It’s for 10.4 but applies to 10.5 almost perfectly.
2. Before starting the vpn server, though, run:

   sudo racoon -vF

   If you’ve done everything right in step 1, the key manager will pop up. Click Always Allow and then ctrl-c to stop racoon running. Now you can start vpnd.

3. It looks like Back To My Mac uses L2TP as well, but changes the default configuration such that it won’t work with normal L2TP clients. If you’re not going to use Back To My Mac again, go into /etc/racoon/remote. You should find a file called anonymous.conf.orig. Copy that file on top of anonymous.conf, killall vpnd and racoon, and then start vpnd up again.
4. You only need to forward UDP ports 500 and 4500 from your firewall. Some other guides recommend forwarding the L2TP port – you shouldn’t do this as you want everything to run through IPSec. You do not need to change the OS X firewall – any of the three settings allows ports 500/4500 through.
5. It looks like when you create a new user, it doesn’t (always?) set up a shadow hash that’s compatible with the MSCHAP2 authentication algorithm. You might have this problem if some users authenticate fine and others don’t. There’s a comment here that tells you what you need to do to fix that.

I made a group named “vpn” using the Accounts preference pane, assigned the users I wanted to have VPN access to it, and then specified it in the com.apple.RemoteAccessServers.plist file instead of using “admin”.

If it’s not working for you, sudo killall vpn racoon, and then start up sudo racoon -vF and try connecting. That way you’ll get a whole heap of debug information on what’s going on with the authentication process, or if indeed the connection is even reaching your computer.

All works brilliantly here. Can connect to the VPN via another Mac or using the iPhone.