Be more careful about marking buffers dirty