pidl s4::NDR::Parser: read hex numbers as numbers for ranges
authorDouglas Bagnall <douglas.bagnall@catalyst.net.nz>
Sat, 30 Nov 2019 10:37:08 +0000 (23:37 +1300)
committerAndrew Bartlett <abartlet@samba.org>
Tue, 10 Dec 2019 02:53:35 +0000 (02:53 +0000)
commit2765b5c1a27232b990537415718e98449617641b
tree02d7bcc06197837cca432e9f9d841794ce40f795
parentefef4366f18515ec78cb025928c20fb7986cd98f
pidl s4::NDR::Parser: read hex numbers as numbers for ranges

Hex numbers in IDL are not parsed as numbers, resulting in warnings
like

Argument 0x2000 isn't numeric in numeric lt (<) at /home/douglas/src/samba/pidl/lib/Parse/Pidl/Samba4/NDR/Parser.pm line 981

not to mention problematic code. We add a utility function to convert
these numbers to numbers.

A typical difference this makes is:

 --- old/default/librpc/gen_ndr/ndr_dcerpc.c     2019-11-30 23:40:32.915816967 +1300
 +++ new/default/librpc/gen_ndr/ndr_dcerpc.c     2019-11-30 17:00:09.055733660 +1300
 @@ -1893,7 +1893,7 @@
         if (ndr_flags & NDR_SCALARS) {
                 NDR_CHECK(ndr_pull_align(ndr, 4));
                 NDR_CHECK(ndr_pull_uint32(ndr, NDR_SCALARS, &r->ReceiveWindowSize));
 -               if (r->ReceiveWindowSize > 0x40000) {
 +               if (r->ReceiveWindowSize < 8192 || r->ReceiveWindowSize > 262144) {
                         return ndr_pull_error(ndr, NDR_ERR_RANGE, "value out of range");
                 }
                 NDR_CHECK(ndr_pull_trailer_align(ndr, 4));

Where the minimum ("0x2000" == 8192) was read as a string, thus
treated as zero.

The treatment as zero was introduced in 142b2a61f8a77b3065ce4c78b459ab714d6d190a
accidentially, which shows why warnings are important.

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
pidl/lib/Parse/Pidl/Samba4/NDR/Parser.pm
pidl/lib/Parse/Pidl/Util.pm