spi: sun4i: disable clocks in the remove function