diff options
| -rw-r--r-- | base/allocator/README | 56 | ||||
| -rw-r--r-- | base/allocator/README.md | 106 | ||||
| -rw-r--r-- | base/allocator/allocator.gyp | 6 |
3 files changed, 109 insertions, 59 deletions
diff --git a/base/allocator/README b/base/allocator/README deleted file mode 100644 index 8a5595f..0000000 --- a/base/allocator/README +++ /dev/null @@ -1,56 +0,0 @@ -Notes about the Chrome memory allocator. - -Background ----------- -We use this library as a generic way to fork into any of several allocators. -Currently we can, at runtime, switch between: - the default windows allocator - the windows low-fragmentation-heap - tcmalloc - -The mechanism for hooking LIBCMT in windows is rather tricky. The core -problem is that by default, the windows library does not declare malloc and -free as weak symbols. Because of this, they cannot be overriden. To work -around this, we start with the LIBCMT.LIB, and manually remove all allocator -related functions from it using the visual studio library tool. Once removed, -we can now link against the library and provide custom versions of the -allocator related functionality. - - -Source code ------------ -This directory contains just the allocator (i.e. shim) layer that switches -between the different underlying memory allocation implementations. - -The tcmalloc library originates outside of Chromium and exists in -../../third_party/tcmalloc (currently, the actual location is defined in the -allocator.gyp file). The third party sources use a vendor-branch SCM pattern to -track Chromium-specific changes independently from upstream changes. - -The general intent is to push local changes upstream so that over -time we no longer need any forked files. - - -Adding a new allocator ----------------------- -Adding a new allocator requires definition of the following five functions: - - extern "C" { - bool init(); - void* malloc(size_t s); - void* realloc(void* p, size_t s); - void free(void* s); - size_t msize(void* p); - } - -All other allocation related functions (new/delete/calloc/etc) have been -implemented generically to work across all allocators. - - -Usage ------ -You can use the different allocators by setting the environment variable -CHROME_ALLOCATOR to: - "tcmalloc" - TC Malloc (default) - "winheap" - Windows default heap - "winlfh" - Windows Low-Fragmentation heap diff --git a/base/allocator/README.md b/base/allocator/README.md new file mode 100644 index 0000000..968e6b0 --- /dev/null +++ b/base/allocator/README.md @@ -0,0 +1,106 @@ +This document describes how malloc / new calls are routed in the various Chrome +platforms. + +Bare in mind that the chromium codebase does not always just use `malloc()`. +Some examples: + - Large parts of the renderer (Blink) use two home-brewed allocators, + PartitionAlloc and BlinkGC (Oilpan). + - Some subsystems, such as the V8 JavaScript engine, handle memory management + autonomously. + - Various parts of the codebase use abstractions such as `SharedMemory` or + `DiscardableMemory` which, similarly to the above, have their own page-level + memory management. + +Background +---------- +The `allocator` target defines at compile-time the platform-specific choice of +the allocator and extra-hooks which services calls to malloc/new. The relevant +build-time flags involved are `use_allocator` and `win_use_allocator_shim`. + +The default choices are as follows: + +**Windows** +`use_allocator: winheap`, the default Windows heap. +Additionally, `static_library` (i.e. non-component) builds have a shim +layer wrapping malloc/new, which is controlled by `win_use_allocator_shim`. +The shim layer provides extra security features, such as preventing large +allocations that can hit signed vs. unsigned bugs in third_party code. + +**Linux Desktop / CrOS** +`use_allocator: tcmalloc`, a forked copy of tcmalloc which resides in +`third_party/tcmalloc/chromium`. Setting `use_allocator: none` causes the build +to fall back to the system (Glibc) symbols. + +**Android** +`use_allocator: none`, always use the allocator symbols coming from Android's +libc (Bionic). As it is developed as part of the OS, it is considered to be +optimized for small devices and more memory-efficient than other choices. +The actual implementation backing malloc symbols in Bionic is up to the board +config and can vary (typically *dlmalloc* or *jemalloc* on most Nexus devices). + +**Mac/iOS** +`use_allocator: none`, we always use the system's allocator implementation. + +In addition, when building for `asan` / `msan` / `syzyasan` `valgrind`, the +both the allocator and the shim layer are disabled. + +Layering and build deps +----------------------- +The `allocator` target provides both the source files for tcmalloc (where +applicable) and the linker flags required for the Windows shim layer. +The `base` target is (almost) the only one depending on `allocator`. No other +targets should depend on it, with the exception of the very few executables / +dynamic libraries that don't depend, either directly or indirectly, on `base` +within the scope of a linker unit. + +More importantly, **no other place outside of `/base` should depend on the +specific allocator** (e.g., directly include `third_party/tcmalloc`). +If such a functional dependency is required that should be achieved using +abstractions in `base` (see `/base/allocator/allocator_extension.h` and +`/base/memory/`) + +**Why `base` depends on `allocator`?** +Because it needs to provide services that depend on the actual allocator +implementation. In the past `base` used to pretend to be allocator-agnostic +and get the dependencies injected by other layers. This ended up being an +inconsistent mess. +See the [allocator cleanup doc][url-allocator-cleanup] for more context. + +Linker unit targets (executables and shared libraries) that depend in some way +on `base` (most of the targets in the codebase) get automatically the correct +set of linker flags to pull in tcmalloc or the Windows shim-layer. + + +Source code +----------- +This directory contains just the allocator (i.e. shim) layer that switches +between the different underlying memory allocation implementations. + +The tcmalloc library originates outside of Chromium and exists in +`../../third_party/tcmalloc` (currently, the actual location is defined in the +allocator.gyp file). The third party sources use a vendor-branch SCM pattern to +track Chromium-specific changes independently from upstream changes. + +The general intent is to push local changes upstream so that over +time we no longer need any forked files. + + +Appendixes +---------- +**How does the Windows shim layer replace the malloc symbols?** +The mechanism for hooking LIBCMT in Windows is rather tricky. The core +problem is that by default, the Windows library does not declare malloc and +free as weak symbols. Because of this, they cannot be overridden. To work +around this, we start with the LIBCMT.LIB, and manually remove all allocator +related functions from it using the visual studio library tool. Once removed, +we can now link against the library and provide custom versions of the +allocator related functionality. +See the script `preb_libc.py` in this folder. + +Related links +------------- +- [Allocator Cleanup Doc - Jan 2016][url-allocator-cleanup] +- [Proposal to use PartitionAlloc as default allocator](https://crbug.com/339604) +- [Memory-Infra: Tools to profile memory usage in Chrome](components/tracing/docs/memory_infra.md) + +[url-allocator-cleanup]: https://docs.google.com/document/d/1V77Kgp_4tfaaWPEZVxNevoD02wXiatnAv7Ssgr0hmjg/edit?usp=sharing diff --git a/base/allocator/allocator.gyp b/base/allocator/allocator.gyp index 402d6de..727f32a 100644 --- a/base/allocator/allocator.gyp +++ b/base/allocator/allocator.gyp @@ -22,9 +22,9 @@ 'disable_debugallocation%': 0, }, 'targets': [ - # Only executables and not libraries should depend on the - # allocator target; only the application (the final executable) - # knows what allocator makes sense. + # The only targets that should depend on allocator are 'base' and + # executables that don't depend, directly or indirectly, on base (a few). + # All the other targets get a transitive dependency on this target via base. { 'target_name': 'allocator', 'variables': { |