net/mlx5: Only cancel recovery work when cleaning up device