It's kind of a blog if you don't think about it too hard
May 3, 2015
This post is for me to look back on later and remember how to fix this!
I installed Ubuntu MATE 14.04 in VirtualBox, installed the guest additions, then switched on Compiz. It worked fine, until I changed the guest screen resolution. Then the guest screen looked like this:
After some trial and error, the fix: in CompizConfig Settings Manager, OpenGL plugin, uncheck "Framebuffer object".