usb-serial: Fix edgeport regression on non-EPiC devices