2018-08 Update: all the networking stuff here is still valid for OpenWrt 18.06.0 (which you should use by preference!). The 64-bit version now works. Similar, more current post: OpenWrt 18.06.0 in VirtualBox, although you'll be back to this post for the networking setup.
Getting OpenWRT to work in VirtualBox was a bit of a challenge. The first was overcoming the idea that I needed an OpenWRT installer. There aren't any: instead there are lots of .img files. (For this, get openwrt-x86-generic-combined-ext4.img.gz from https://downloads.openwrt.org/barrier_breaker/14.07/x86/generic/ .) They're not installers: they're disk images, which is different. An installer boots on its own, and you then point it at a location to be installed. A disk image requires some other form of software that's able to write it onto the target disk. With OpenWRT you usually use the old router software, telling it where the new firmware is: it downloads and unpacks it. With OpenWRT, the command you need is:
VBoxManage convertfromraw --format VDI openwrt-x86-generic-combined-ext4.img openwrt-BB-x86.vdi
Note that I'm using the image for x86 - important, as most of the images are for embedded chips of various types with x86 being the odd man out. I'm also using the 32-bit version as I had problems with the 64-bit . The above command writes the OpenWRT image as a VDI file, which is now bootable. It's possible to do this without uncompressing the gzipped image first. 
The first problem you're likely to notice after booting is that you don't have a network. OpenWRT assumes it's 192.168.1.1/24, which is rarely true with VirtualBox. I've solved the problem in two ways:
# uci show network # doesn't change anything # uci set network.lan.ipaddr=10.0.2.15 # uci set network.lan.netmask=255.255.255.0 # uci commit # this updates file /etc/config/network # ifdown lan # ifup lan
Edit /etc/config/network and find this line: option ipaddr '192.168.1.1' and change it to option ipaddr '10.0.2.15'. Then run /etc/init.d/network reload.
In either case, you should be on the local network (you can test with ping 10.0.2.2, which is VirtualBox's gateway). But this doesn't take care of your routing or DNS.
Keep in mind that what I'm doing here is creating breaking changes in the standard behaviour of OpenWRT: this is for fixed IP addresses and fixed routes because I'm running OpenWRT in VirtualBox - a very strange circumstance.
This works, although only temporarily:
route add default gw 10.0.2.2 br-lan
That tells the machine that 10.0.2.2 is the default way out to the world, and that it should access it through the br-lan interface. Putting this line in /etc/rc.local should make it permanent through a reboot, but doesn't - presumably because the network is brought up after rc.local is run. So, the fix is to edit /etc/config/network again, and add this stanza at the end:
config 'route' option 'interface' 'lan' option 'target' '*' option 'netmask' '255.255.255.0' option 'gateway' '172.16.0.1'
Where '172.16.0.1' is your home gateway/router.
One more step. vim /etc/resolv.conf:
search lan #nameserver 127.0.0.1 nameserver 172.16.0.1 nameserver 18.104.22.168
The commented-out line is the default - OpenWRT assumes it's the router, and is therefore working as a passthrough DNS server (not sure if that's the correct term). Since it's in VirtualBox, that's not true. You need to add a valid DNS - usually your router IP, and I've also added 22.214.171.124, one of Google's public routers. If you're not sure of your nameserver address, go to a console in the host machine and type
cat /etc/resolv.conf | grep nameserver on any Unixy machine, or type ipconfig /all on a Windows-based machine and look for "DNS Servers."
This isn't permanent because resolv.conf is a link to /tmp/resolv.conf and is replaced at reboot. The solution is to delete the link and replace /etc/resolv.conf with a real file as above.
SSH on OpenWRT is handled by Dropbear, and that means you have to deal with its idiosyncrasies. This means copying keys not to ~/.ssh/known_hosts but instead to /etc/dropbear/known_hosts. And the keys apparently cannot be ecdsa. rsa works: I'm not sure about other protocols.
Differences from Router Hardware
As was mentioned in passing near the beginning, OpenWRT is mostly oriented to the idea of a block of flash memory where the image is written. On an actual router, this memory is treated as read-only from then on, and new packages are installed in an /overlay/ folder. The i686 build used here treats everything as writable, and doesn't store anything in the /overlay/ folder. The squashfs image(s?) in the download directory seem to behave in this manner if you prefer an experience more similar to regular router hardware.
- https://www.youtube.com/watch?v=Z13R6m4nYzI - installation - too long, but credit where it's due: I had no idea where to start before I saw this
- https://forum.openwrt.org/viewtopic.php?id=16962 - basic static routes
- my OpenWRT Notes
- my VirtualBox Notes
|||During boot the 64-bit version, regardless of the settings I experimented with, always stopped entirely at "Switched to clocksource tsc" and never went any further. I did some research - it appears to be a relatively common stop-point for Linux kernels, but I didn't find a fix. Yes, my host machine is 64-bit. Update: "Chaos Calmer RC3" 64-bit boots without issues where CC RC2 and BB did not.|
|||If you don't want to uncompress the image: run gunzip -l openwrt-x86-generic-combined-ext4.img.gz to get the uncompressed file size, then run gunzip --stdout openwrt-x86-generic-combined-ext4.img.gz | VBoxManage convertfromraw --format VDI stdin 55050240 openwrt-BB-x86.vdi (where "55050240" was the correct size for the "Barrier Breaker" image I was working with).|