block: cache current nsec time in struct blk_plug

Querying the current time is the most costly thing we do in the block
layer per IO, and depending on kernel config settings, we may do it
many times per IO.

None of the callers actually need nsec granularity. Take advantage of
that by caching the current time in the plug, with the assumption here
being that any time checking will be temporally close enough that the
slight loss of precision doesn't matter.

If the block plug gets flushed, eg on preempt or schedule out, then
we invalidate the cached clock.

On a basic peak IOPS test case with iostats enabled, this changes
the performance from:

IOPS=108.41M, BW=52.93GiB/s, IOS/call=31/31
IOPS=108.43M, BW=52.94GiB/s, IOS/call=32/32
IOPS=108.29M, BW=52.88GiB/s, IOS/call=31/32
IOPS=108.35M, BW=52.91GiB/s, IOS/call=32/32
IOPS=108.42M, BW=52.94GiB/s, IOS/call=31/31
IOPS=108.40M, BW=52.93GiB/s, IOS/call=32/32
IOPS=108.31M, BW=52.89GiB/s, IOS/call=32/31

to

IOPS=118.79M, BW=58.00GiB/s, IOS/call=31/32
IOPS=118.62M, BW=57.92GiB/s, IOS/call=31/31
IOPS=118.80M, BW=58.01GiB/s, IOS/call=32/31
IOPS=118.78M, BW=58.00GiB/s, IOS/call=32/32
IOPS=118.69M, BW=57.95GiB/s, IOS/call=32/31
IOPS=118.62M, BW=57.92GiB/s, IOS/call=32/31
IOPS=118.63M, BW=57.92GiB/s, IOS/call=31/32

which is more than a 9% improvement in performance. Looking at perf diff,
we can see a huge reduction in time overhead:

    10.55%     -9.88%  [kernel.vmlinux]  [k] read_tsc
     1.31%     -1.22%  [kernel.vmlinux]  [k] ktime_get

Note that since this relies on blk_plug for the caching, it's only
applicable to the issue side. But this is where most of the time calls
happen anyway. On the completion side, cached time stamping is done with
struct io_comp patch, as long as the driver supports it.

It's also worth noting that the above testing doesn't enable any of the
higher cost CPU items on the block layer side, like wbt, cgroups,
iocost, etc, which all would add additional time querying and hence
overhead. IOW, results would likely look even better in comparison with
those enabled, as distros would do.

Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/block/blk-core.c b/block/blk-core.c
index de771093b52687ae2431af36bf75b73ccaa1bbf0..13b449df5ba02bdf18fe685c4d413f9639acda5f 100644 (file)
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1083,6 +1083,7 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios)
        if (tsk->plug)
                return;
 
+       plug->cur_ktime = 0;
        plug->mq_list = NULL;
        plug->cached_rq = NULL;
        plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT);

diff --git a/block/blk.h b/block/blk.h
index 79ae533cdf026f8c592326e063ac75be5239fbea..14bbc4b780f275cd175e26f38f9f97278fb54ddd 100644 (file)
--- a/block/blk.h
+++ b/block/blk.h
@@ -519,7 +519,19 @@ static inline int req_ref_read(struct request *req)
 
 static inline u64 blk_time_get_ns(void)
 {
-       return ktime_get_ns();
+       struct blk_plug *plug = current->plug;
+
+       if (!plug)
+               return ktime_get_ns();
+
+       /*
+        * 0 could very well be a valid time, but rather than flag "this is
+        * a valid timestamp" separately, just accept that we'll do an extra
+        * ktime_get_ns() if we just happen to get 0 as the current time.
+        */
+       if (!plug->cur_ktime)
+               plug->cur_ktime = ktime_get_ns();
+       return plug->cur_ktime;
 }
 
 static inline ktime_t blk_time_get(void)

diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 99e4f5e722132c2c4f301816bbab7871f2f2ccb0..996d2ad756ff782e8f2e65cc63f588325d2f0301 100644 (file)
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -942,6 +942,7 @@ struct blk_plug {
 
        /* if ios_left is > 1, we can batch tag/rq allocations */
        struct request *cached_rq;
+       u64 cur_ktime;
        unsigned short nr_ios;
 
        unsigned short rq_count;