Fix some more "char is unsigned" issues, and a possible "char is signed" one.

Fix some more "char is unsigned" issues, and a possible "char is signed" one.

C neither guarantees that char is signed nor that it's unsigned.  Make
the str_to_nibble tables arrays of gint8, to make sure they can hold
numbers between 0 and 15 as well as -1.  Cast gchar to guchar, not int,
when using it as a subscript into that array, so that the subscripts are
in the range 0 to 255, not -128 to 127.

Change-Id: Ib85de5aa4e83ae9efd808c78ce3f86f45b4a3f2a
Reviewed-on: https://code.wireshark.org/review/4734
Reviewed-by: Guy Harris <guy@alum.mit.edu>