Fix some more leaks in error returns from open routines (if the open can

Fix some more leaks in error returns from open routines (if the open can
fail after the private data is allocated, you have to free the private
data).

The file header in nettl files is 128 bytes - use a #define for it, and
also a #define for the magic number size.

git-svn-id: http://anonsvn.wireshark.org/wireshark/trunk@14553 f5534014-38df-0310-8fa8-9805f1628bb7