ftrace: Use asynchronous grace period for register_ftrace_direct()

When running heavy test workloads with KASAN enabled, RCU Tasks grace
periods can extend for many tens of seconds, significantly slowing
trace registration.  Therefore, make the registration-side RCU Tasks
grace period be asynchronous via call_rcu_tasks().

Link: https://lore.kernel.org/linux-trace-kernel/ac05be77-2972-475b-9b57-56bef15aa00a@paulmck-laptop
Reported-by: Jakub Kicinski <kuba@kernel.org>
Reported-by: Alexei Starovoitov <ast@kernel.org>
Reported-by: Chris Mason <clm@fb.com>
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>

diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 3951c0d0ec631b18d73f0edb662702009e717198..f9223513414a3487d649a0c4237b4dd6c014eae5 100644 (file)
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5366,6 +5366,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
        }
 }
 
+static void register_ftrace_direct_cb(struct rcu_head *rhp)
+{
+       struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
+
+       free_ftrace_hash(fhp);
+}
+
 /**
  * register_ftrace_direct - Call a custom trampoline directly
  * for multiple functions registered in @ops
@@ -5464,10 +5471,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
  out_unlock:
        mutex_unlock(&direct_mutex);
 
-       if (free_hash && free_hash != EMPTY_HASH) {
-               synchronize_rcu_tasks();
-               free_ftrace_hash(free_hash);
-       }
+       if (free_hash && free_hash != EMPTY_HASH)
+               call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
 
        if (new_hash)
                free_ftrace_hash(new_hash);