5 The 'emem' memory manager (described in README.malloc) has been a part of
6 Wireshark since 2005 and has served us well, but is starting to show its age.
7 The framework has become increasingly difficult to maintain, and limitations
8 in the API have blocked progress on other long-term goals such as multi-
9 threading, and opening multiple files at once.
11 The 'wmem' memory manager is a new memory management framework that replaces
12 emem. It provides a significantly updated API, a more modular design, and it
13 isn't all jammed into one 2500-line file.
15 Wmem was originally conceived in this email to the wireshark-dev mailing list:
16 https://www.wireshark.org/lists/wireshark-dev/201210/msg00178.html
18 The wmem code can now be found in epan/wmem/ in the Wireshark source tree.
20 2. Usage for Consumers
22 If you're writing a dissector, or other "userspace" code, then using wmem
23 should be very similar to using emem. All you need to do is include the header
24 (epan/wmem/wmem.h) and get a handle to a memory pool (if you want to *create*
25 a memory pool, see the section "3. Usage for Producers" below).
27 A memory pool is an opaque pointer to an object of type wmem_allocator_t, and
28 it is the very first parameter passed to almost every call you make to wmem.
29 Other than that parameter (and the fact that functions are prefixed wmem_
30 instead of ep_ or se_) usage is exactly like that of emem. For example:
32 wmem_alloc(myPool, 20);
34 allocates 20 bytes in the pool pointed to by myPool.
38 2.1.1 (Sort Of) Global Pools
40 Dissectors that include the wmem header file will have three pools available
41 to them automatically: wmem_packet_scope(), wmem_file_scope() and
44 The packet pool is scoped to the dissection of each packet, replacing
45 emem's ep_ allocators. The file pool is scoped to the dissection of each file,
46 replacing emem's se_ allocators. For example:
49 se_malloc(sizeof(guint));
51 could be replaced with
53 wmem_alloc(wmem_packet_scope(), 32);
54 wmem_alloc(wmem_file_scope(), sizeof(guint));
56 NB: Using these pools outside of the appropriate scope (eg using the packet
57 pool when there isn't a packet being dissected) will throw an assertion.
58 See the comment in epan/wmem/wmem_scopes.c for details.
60 The epan pool is scoped to the library's lifetime - memory allocated in it is
61 not freed until epan_cleanup() is called, which is typically at the end of the
66 Certain places (such as AT_STRINGZ address allocations) need their memory to
67 stay around a little longer than the usual packet scope - basically until the
68 next packet is dissected. This is effectively the scope of Wireshark's pinfo
69 structure, so the pinfo struct has a 'pool' member which is a wmem pool scoped
70 to the lifetime of the pinfo struct.
74 Full documentation for each function (parameters, return values, behaviours)
75 lives (or will live) in Doxygen-format in the header files for those functions.
76 This is just an overview of which header files you should be looking at.
81 - Basic memory management functions like malloc, realloc and free.
86 - Utility functions for manipulating null-terminated C-style strings.
87 Functions like strdup and strdup_printf.
90 - A managed string object implementation, similar to std::string in C++ or
93 2.2.3 Container Data Structures
96 - A growable array (AKA vector) implementation.
99 - A doubly-linked list implementation.
102 - A queue implementation (first-in, first-out).
105 - A stack implementation (last-in, first-out).
108 - A balanced binary tree (red-black tree) implementation.
110 2.2.4 Miscellanious Utilities
113 - Misc. utility functions like memdup.
117 WARNING: You probably don't actually need these; use them only when you're
118 sure you understand the dangers.
120 Sometimes (though hopefully rarely) it may be necessary to store data in a wmem
121 pool that requires additional cleanup before it is freed. For example, perhaps
122 you have a pointer to a file-handle that needs to be closed. In this case, you
123 can register a callback with the wmem_register_cleanup_callback function
124 declared in wmem_user_cb.h. Every time the memory in a pool is freed, all
125 registered cleanup functions are called first.
127 Note that callback calling order is not defined, you cannot rely on a
128 certain callback being called before or after another.
130 WARNING: Manually freeing or moving memory (with wmem_free or wmem_realloc)
131 will NOT trigger any callbacks. It is an error to call either of
132 those functions on memory if you have a callback registered to deal
133 with the contents of that memory.
135 3. Usage for Producers
137 NB: If you're just writing a dissector, you probably don't need to read
140 One of the problems with the old emem framework was that there were basically
141 two allocator backends (glib and mmap) that were all mixed together in a mess
142 of if statements, environment variables and #ifdefs. In wmem the different
143 allocator backends are cleanly separated out, and it's up to the owner of the
146 3.1 Available Allocator Back-Ends
148 Each available allocator type has a corresponding entry in the
149 wmem_allocator_type_t enumeration defined in wmem_core.h. See the doxygen
150 comments in that header file for details on each type.
154 To create a pool, include the regular wmem header and call the
155 wmem_allocator_new() function with the appropriate type value.
158 #include "wmem/wmem.h"
160 wmem_allocator_t *myPool;
161 myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
163 From here on in, you don't need to remember which type of allocator you used
164 (although allocator authors are welcome to expose additional allocator-specific
165 helper functions in their headers). The "myPool" variable can be passed around
166 and used as normal in allocation requests as described in section 2 of this
169 3.3 Destroying a Pool
171 Regardless of which allocator you used to create a pool, it can be destroyed
172 with a call to the function wmem_destroy_allocator(). For example:
174 #include "wmem/wmem.h"
176 wmem_allocator_t *myPool;
178 myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
180 /* Allocate some memory in myPool ... */
182 wmem_destroy_allocator(myPool);
184 Destroying a pool will free all the memory allocated in it.
188 It is possible to free all the memory in a pool without destroying it,
189 allowing it to be reused later. Depending on the type of allocator, doing this
190 (by calling wmem_free_all()) can be significantly cheaper than fully destroying
191 and recreating the pool. This method is therefore recommended, especially when
192 the pool would otherwise be scoped to a single iteration of a loop. For example:
194 #include "wmem/wmem.h"
196 wmem_allocator_t *myPool;
198 myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
201 /* Allocate some memory in myPool ... */
203 /* Free the memory, faster than destroying and recreating
204 the pool each time through the loop. */
205 wmem_free_all(myPool);
207 wmem_destroy_allocator(myPool);
211 Despite being written in Wireshark's standard C90, wmem follows a fairly
212 object-oriented design pattern. Although efficiency is always a concern, the
213 primary goals in writing wmem were maintainability and preventing memory
216 4.1 struct _wmem_allocator_t
218 The heart of wmem is the _wmem_allocator_t structure defined in the
219 wmem_allocator.h header file. This structure uses C function pointers to
220 implement a common object-oriented design pattern known as an interface (also
221 known as an abstract class to those who are more familiar with C++).
223 Different allocator implementations can provide exactly the same interface by
224 assigning their own functions to the members of an instance of the structure.
225 The structure has eight members in three groups.
227 4.1.1 Implementation Details
232 The private_data pointer is a void pointer that the allocator implementation can
233 use to store whatever internal structures it needs. A pointer to private_data is
234 passed to almost all of the other functions that the allocator implementation
237 The type field is an enumeration of type wmem_allocator_type_t (see
238 section 3.1). Its value is set by the wmem_allocator_new() function, not
239 by the implementation-specific constructor. This field should be considered
240 read-only by the allocator implementation.
242 4.1.2 Consumer Functions
248 These function pointers should be set to functions with semantics obviously
249 similar to their standard-library namesakes. Each one takes an extra parameter
250 that is a copy of the allocator's private_data pointer.
252 Note that realloc() and free() are not expected to be called directly by user
253 code in most cases - they are primarily optimisations for use by data
254 structures that wmem might want to implement (it's hard, for example, to
255 implement a dynamically sized array without some form of realloc).
257 Also note that allocators do not have to handle NULL pointers or 0-length
258 requests in any way - those checks are done in an allocator-agnostic way
259 higher up in wmem. Allocator authors can assume that all incoming pointers
260 (to realloc and free) are non-NULL, and that all incoming lengths (to malloc
261 and realloc) are non-0.
263 4.1.3 Producer/Manager Functions
269 All of these functions take only one parameter, which is the allocator's
270 private_data pointer.
272 The free_all() function should free all the memory currently allocated in the
273 pool. Note that this is not necessarily exactly the same as calling free()
274 on all the allocated blocks - free_all() is allowed to do additional cleanup
275 or to make use of optimizations not available when freeing one block at a time.
277 The gc() function should do whatever it can to reduce excess memory usage in
278 the dissector by returning unused blocks to the OS, optimizing internal data
281 The cleanup() function should do any final cleanup and free any and all memory.
282 It is basically the equivalent of a destructor function. For simplicity, wmem
283 is guaranteed to call free_all() immediately before this function. There is no
284 such guarantee that gc() has (ever) been called.
286 4.2 Pool-Agnostic API
288 One of the issues with emem was that the API (including the public data
289 structures) required wrapper functions for each scope implemented. Even
290 if there was a stack implementation in emem, it wasn't necessarily available
291 for use with file-scope memory unless someone took the time to write se_stack_
292 wrapper functions for the interface.
294 In wmem, all public APIs take the pool as the first argument, so that they can
295 be written once and used with any available memory pool. Data structures like
296 wmem's stack implementation only take the pool when created - the provided
297 pointer is stored internally with the data structure, and subsequent calls
298 (like push and pop) will take the stack itself instead of the pool.
302 The primary debugging control for wmem is the WIRESHARK_DEBUG_WMEM_OVERRIDE
303 environment variable. If set, this value forces all calls to
304 wmem_allocator_new() to return the same type of allocator, regardless of which
305 type is requested normally by the code. It currently has three valid values:
307 - The value "simple" forces the use of WMEM_ALLOCATOR_SIMPLE. The valgrind
308 script currently sets this value, since the simple allocator is the only
309 one whose memory allocations are trackable properly by valgrind.
311 - The value "strict" forces the use of WMEM_ALLOCATOR_STRICT. The fuzz-test
312 script currently sets this value, since the goal when fuzz-testing is to find
313 as many errors as possible.
315 - The value "block" forces the use of WMEM_ALLOCATOR_BLOCK. This is not
316 currently used by any scripts, but is useful for stress-testing the block
319 Note that regardless of the value of this variable, it will always be safe to
320 call allocator-specific helpers functions. They are required to be safe no-ops
321 if the allocator argument is of the wrong type.
325 There is a simple test suite for wmem that lives in the file wmem_test.c and
326 should get automatically built into the binary 'wmem_test' when building
327 Wireshark. It contains at least basic tests for all existing functionality.
328 The suite is run automatically by the build-bots via the shell script
329 test/test.sh which calls out to test/suite-unittests.sh.
331 New features added to wmem (allocators, data structures, utility
332 functions, etc.) must also have tests added to this suite.
334 The test suite could potentially use a clean-up by someone more
335 intimately familiar with Glib's testing framework, but it does the job.
339 The following is a list of things that emem didn't provide but that it might
346 * Editor modelines - http://www.wireshark.org/tools/modelines.html
351 * indent-tabs-mode: nil
354 * vi: set shiftwidth=4 tabstop=8 expandtab:
355 * :indentSize=4:tabSize=8:noTabs=true: