sparc64: optimize struct page zeroing
authorPavel Tatashin <>
Thu, 16 Nov 2017 01:36:48 +0000 (17:36 -0800)
committerLinus Torvalds <>
Thu, 16 Nov 2017 02:21:05 +0000 (18:21 -0800)
sparc64: optimize struct page zeroing

Add an optimized mm_zero_struct_page(), so struct page's are zeroed
without calling memset().  We do eight to ten regular stores based on
the size of struct page.  Compiler optimizes out the conditions of
switch() statement.

SPARC-M6 with 15T of memory, single thread performance:

                               BASE            FIX  OPTIMIZED_FIX
        bootmem_init   28.440467985s   2.305674818s   2.305161615s
free_area_init_nodes  202.845901673s 225.343084508s 172.556506560s
Total                 231.286369658s 227.648759326s 174.861668175s

BASE:  current linux
FIX:   This patch series without "optimized struct page zeroing"
OPTIMIZED_FIX: This patch series including the current patch.

bootmem_init() is where memory for struct pages is zeroed during
allocation.  Note, about two seconds in this function is a fixed time:
it does not increase as memory is increased.

Signed-off-by: Pavel Tatashin <>
Reviewed-by: Steven Sistare <>
Reviewed-by: Daniel Jordan <>
Reviewed-by: Bob Picco <>
Acked-by: David S. Miller <>
Cc: Alexander Potapenko <>
Cc: Andrey Ryabinin <>
Cc: Ard Biesheuvel <>
Cc: Catalin Marinas <>
Cc: Christian Borntraeger <>
Cc: Dmitry Vyukov <>
Cc: Heiko Carstens <>
Cc: "H. Peter Anvin" <>
Cc: Ingo Molnar <>
Cc: Mark Rutland <>
Cc: Matthew Wilcox <>
Cc: Mel Gorman <>
Cc: Michal Hocko <>
Cc: Michal Hocko <>
Cc: Sam Ravnborg <>
Cc: Thomas Gleixner <>
Cc: Will Deacon <>
Signed-off-by: Andrew Morton <>
Signed-off-by: Linus Torvalds <>