5 NB: Wmem still does not provide all of the functionality of emem
6 (see README.malloc), although it should provide most of it. New code
7 may still need to use emem for the time being.
9 The 'emem' memory manager (described in README.malloc) has been a part of
10 Wireshark since 2005 and has served us well, but is starting to show its age.
11 The framework has become increasingly difficult to maintain, and limitations
12 in the API have blocked progress on other long-term goals such as multi-
13 threading, and opening multiple files at once.
15 The 'wmem' memory manager is an attempt to write a new memory management
16 framework to replace emem. It provides a significantly updated API, a more
17 modular design, and it isn't all jammed into one 2500-line file.
19 Wmem was originally conceived in this email to the wireshark-dev mailing list:
20 https://www.wireshark.org/lists/wireshark-dev/201210/msg00178.html
22 The wmem code can now be found in epan/wmem/ in the Wireshark source tree.
24 2. Usage for Consumers
26 If you're writing a dissector, or other "userspace" code, then using wmem
27 should be very similar to using emem. All you need to do is include the header
28 (epan/wmem/wmem.h) and get a handle to a memory pool (if you want to *create*
29 a memory pool, see the section "3. Usage for Producers" below).
31 A memory pool is an opaque pointer to an object of type wmem_allocator_t, and
32 it is the very first parameter passed to almost every call you make to wmem.
33 Other than that parameter (and the fact that functions are prefixed wmem_
34 instead of ep_ or se_) usage is exactly like that of emem. For example:
36 wmem_alloc(myPool, 20);
38 allocates 20 bytes in the pool pointed to by myPool.
42 2.1.1 (Sort Of) Global Pools
44 Dissectors that include the wmem header file will have three pools available
45 to them automatically: wmem_packet_scope(), wmem_file_scope() and
48 The packet pool is scoped to the dissection of each packet, replacing
49 emem's ep_ allocators. The file pool is scoped to the dissection of each file,
50 replacing emem's se_ allocators. For example:
53 se_malloc(sizeof(guint));
55 could be replaced with
57 wmem_alloc(wmem_packet_scope(), 32);
58 wmem_alloc(wmem_file_scope(), sizeof(guint));
60 NB: Using these pools outside of the appropriate scope (eg using the packet
61 pool when there isn't a packet being dissected) will throw an assertion.
62 See the comment in epan/wmem/wmem_scopes.c for details.
64 The epan pool is scoped to the library's lifetime - memory allocated in it is
65 not freed until epan_cleanup() is called, which is typically at the end of the
70 Certain places (such as AT_STRINGZ address allocations) need their memory to
71 stay around a little longer than the usual packet scope - basically until the
72 next packet is dissected. This is effectively the scope of Wireshark's pinfo
73 structure, so the pinfo struct has a 'pool' member which is a wmem pool scoped
74 to the lifetime of the pinfo struct.
78 Full documentation for each function (parameters, return values, behaviours)
79 lives (or will live) in Doxygen-format in the header files for those functions.
80 This is just an overview of which header files you should be looking at.
85 - Basic memory management functions like malloc, realloc and free.
90 - Utility functions for manipulating null-terminated C-style strings.
91 Functions like strdup and strdup_printf.
94 - A managed string object implementation, similar to std::string in C++ or
97 2.2.3 Container Data Structures
100 - A singly-linked list implementation.
103 - A stack implementation (push, pop, etc).
107 WARNING: You probably don't actually need these; use them only when you're
108 sure you understand the dangers.
110 Sometimes (though hopefully rarely) it may be necessary to store data in a wmem
111 pool that requires additional cleanup before it is freed. For example, perhaps
112 you have a pointer to a file-handle that needs to be closed. In this case, you
113 can register a callback with the wmem_register_cleanup_callback function
114 declared in wmem_user_cb.h. This function takes as parameters:
116 - boolean indicating whether or not the callback should be recurring (ie
117 happens once if FALSE, or every time if TRUE)
118 - the callback function (signature defined in wmem_user_cb.h)
119 - a void user_data pointer
121 Every time the memory in a pool is freed, all registered cleanup functions are
122 called first, being passed:
123 - a pointer to the allocator
124 - a boolean indicating if this is just a free_all (FALSE) or if this was
125 triggered by destruction of the entire pool (TRUE) - mostly useful for
127 - whatever user_data was registered with that callback.
129 Note that the user_data pointer is not freed when a callback is finished, you
130 have to do that yourself in the callback, or just allocate it in the
131 appropriate wmem pool.
133 Also note that callback calling order is not defined, you cannot rely on a
134 certain callback being called before or after another.
136 WARNING: Manually freeing or moving memory (with wmem_free or wmem_realloc)
137 will NOT trigger any callbacks. It is an error to call either of
138 those functions on memory if you have a callback registered to deal
139 with the contents of that memory.
141 3. Usage for Producers
143 NB: If you're just writing a dissector, you probably don't need to read
146 One of the problems with the old emem framework was that there were basically
147 two allocator backends (glib and mmap) that were all mixed together in a mess
148 of if statements, environment variables and #ifdefs. In wmem the different
149 allocator backends are cleanly separated out, and it's up to the owner of the
152 3.1 Available Allocator Back-Ends
154 Each available allocator type has a corresponding entry in the
155 wmem_allocator_type_t enumeration defined in wmem_core.h.
157 The currently available allocators are:
158 - WMEM_ALLOCATOR_SIMPLE (wmem_allocator_simple.*)
159 A trivial allocator that g_allocs requested memory and tracks
160 allocations via a GHashTable. As simple as possible, intended more as
161 a demo than for practical usage. Also has the benefit of being friendly
162 to tools like valgrind.
163 - WMEM_ALLOCATOR_BLOCK (wmem_allocator_block.*)
164 A block allocator that grabs large chunks of memory at a time
165 (8 MB currently) and serves allocations out of those chunks.
166 Designed for efficiency, especially in the free_all operation.
167 - WMEM_ALLOCATOR_STRICT (wmem_allocator_strict.*)
168 An allocator that does its best to find invalid memory usage via
169 things like canaries and scrubbing freed memory. Valgrind is the
170 better choice on platforms that support it.
174 To create a pool, include the regular wmem header and call the
175 wmem_allocator_new() function with the appropriate type value.
178 #include "wmem/wmem.h"
180 wmem_allocator_t *myPool;
181 myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
183 From here on in, you don't need to remember which type of allocator you used
184 (although allocator authors are welcome to expose additional allocator-specific
185 helper functions in their headers). The "myPool" variable can be passed around
186 and used as normal in allocation requests as described in section 2 of this
189 3.3 Destroying a Pool
191 Regardless of which allocator you used to create a pool, it can be destroyed
192 with a call to the function wmem_destroy_allocator(). For example:
194 #include "wmem/wmem.h"
196 wmem_allocator_t *myPool;
198 myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
200 /* Allocate some memory in myPool ... */
202 wmem_destroy_allocator(myPool);
204 Destroying a pool will free all the memory allocated in it.
208 It is possible to free all the memory in a pool without destroying it,
209 allowing it to be reused later. Depending on the type of allocator, doing this
210 (by calling wmem_free_all()) can be significantly cheaper than fully destroying
211 and recreating the pool. This method is therefore recommended, especially when
212 the pool would otherwise be scoped to a single iteration of a loop. For example:
214 #include "wmem/wmem.h"
216 wmem_allocator_t *myPool;
218 myPool = wmem_allocator_new(WMEM_ALLOCATOR_SIMPLE);
221 /* Allocate some memory in myPool ... */
223 /* Free the memory, faster than destroying and recreating
224 the pool each time through the loop. */
225 wmem_free_all(myPool);
227 wmem_destroy_allocator(myPool);
231 Despite being written in Wireshark's standard C90, wmem follows a fairly
232 object-oriented design pattern. Although efficiency is always a concern, the
233 primary goals in writing wmem were maintainability and preventing memory
236 4.1 struct _wmem_allocator_t
238 The heart of wmem is the _wmem_allocator_t structure defined in the
239 wmem_allocator.h header file. This structure uses C function pointers to
240 implement a common object-oriented design pattern known as an interface (also
241 known as an abstract class to those who are more familiar with C++).
243 Different allocator implementations can provide exactly the same interface by
244 assigning their own functions to the members of an instance of the structure.
245 The structure has eight members in three groups.
247 4.1.1 Implementation Details
252 The private_data pointer is a void pointer that the allocator implementation can
253 use to store whatever internal structures it needs. A pointer to private_data is
254 passed to almost all of the other functions that the allocator implementation
257 The type field is an enumeration of type wmem_allocator_type_t (see
258 section 3.1). Its value is set by the wmem_allocator_new() function, not
259 by the implementation-specific constructor. This field should be considered
260 read-only by the allocator implementation.
262 4.1.2 Consumer Functions
268 These function pointers should be set to functions with semantics obviously
269 similar to their standard-library namesakes. Each one takes an extra parameter
270 that is a copy of the allocator's private_data pointer.
272 Note that realloc() and free() are not expected to be called directly by user
273 code in most cases - they are primarily optimisations for use by data
274 structures that wmem might want to implement (it's hard, for example, to
275 implement a dynamically sized array without some form of realloc).
277 Also note that allocators do not have to handle NULL pointers or 0-length
278 requests in any way - those checks are done in an allocator-agnostic way
279 higher up in wmem. Allocator authors can assume that all incoming pointers
280 (to realloc and free) are non-NULL, and that all incoming lengths (to malloc
281 and realloc) are non-0.
283 4.1.3 Producer/Manager Functions
289 The free_all() function takes the private_data pointer and should free all the
290 memory currently allocated in the pool. Note that this is not necessarilly
291 exactly the same as calling free() on all the allocated blocks - free_all() is
292 allowed to do additional cleanup or to make use of optimizations not available
293 when freeing one block at a time.
295 The gc() function takes the private_data pointer and should do whatever it can
296 to reduce excess memory usage in the dissector by returning unused blocks to
297 the OS, optimizing internal data structures, etc.
299 The destroy() function does NOT take the private_data pointer - it instead takes
300 a pointer to the allocator structure as a whole, since that structure may also
301 need freeing. This function can assume that free_all() has been called
302 immediately before it (though it can make no assumptions about whether or not
303 gc() has ever been called).
305 4.2 Pool-Agnostic API
307 One of the issues with emem was that the API (including the public data
308 structures) required wrapper functions for each scope implemented. Even
309 if there was a stack implementation in emem, it wasn't necessarily available
310 for use with file-scope memory unless someone took the time to write se_stack_
311 wrapper functions for the interface.
313 In wmem, all public APIs take the pool as the first argument, so that they can
314 be written once and used with any available memory pool. Data structures like
315 wmem's stack implementation only take the pool when created - the provided
316 pointer is stored internally with the data structure, and subsequent calls
317 (like push and pop) will take the stack itself instead of the pool.
321 The primary debugging control for wmem is the WIRESHARK_DEBUG_WMEM_OVERRIDE
322 environment variable. If set, this value forces all calls to
323 wmem_allocator_new() to return the same type of allocator, regardless of which
324 type is requested normally by the code. It currently has three valid values:
326 - The value "simple" forces the use of WMEM_ALLOCATOR_SIMPLE. The valgrind
327 script currently sets this value, since the simple allocator is the only
328 one whose memory allocations are trackable properly by valgrind.
330 - The value "strict" forces the use of WMEM_ALLOCATOR_STRICT. The fuzz-test
331 script currently sets this value, since the goal when fuzz-testing is to find
332 as many errors as possible.
334 - The value "block" forces the use of WMEM_ALLOCATOR_BLOCK. This is not
335 currently used by any scripts, but is useful for stress-testing the block
338 Note that regardless of the value of this variable, it will always be safe to
339 call allocator-specific helpers functions. They are required to be safe no-ops
340 if the allocator argument is of the wrong type.
344 There is a simple test suite for wmem that lives in the file wmem_test.c and
345 should get automatically built into the binary 'wmem_test' when building
346 Wireshark. It contains at least basic tests for all existing functionality.
347 The suite is run automatically by the build-bots via the shell script
348 test/test.sh which calls out to test/suite-unittests.sh.
350 New features added to wmem (allocators, data structures, utility
351 functions, etc.) must also have tests added to this suite.
353 The test suite could potentially use a clean-up by someone more
354 intimately familiar with Glib's testing framework, but it does the job.
358 The following is an incomplete list of things that emem provides but wmem has
364 The following is a list of things that emem doesn't provide but that it might
365 be nice if wmem did provide them:
372 * Editor modelines - http://www.wireshark.org/tools/modelines.html
377 * indent-tabs-mode: nil
380 * vi: set shiftwidth=4 tabstop=8 expandtab:
381 * :indentSize=4:tabSize=8:noTabs=true: