Minimal Raspbian distribution

Packages that should be installed: 

libraspberrypi-bin
raspberrypi-bootloader
raspi-config

Packages that should be removed: 

lxde-common
lxsession

jwhitehorn/raspbian-mod: script to slim down a Raspbian installation

Building an image from scratch

Buildroot has instructions for an Raspberry Pi target.