netlink: add mask validation