When I gave up climbing for many years, my shoe size went back up at least one full size, and the weird "horn" on the back of my ankle completely dissapeared.
These days I leave my boots a little bigger which has the benefits of not having to take them off every ten minutes too.