btrfs: wait for actual caching progress during allocation

Recently we've been having mysterious hangs while running generic/475 on
the CI system.  This turned out to be something like this:

  Task 1
  dmsetup suspend --nolockfs
  -> __dm_suspend
   -> dm_wait_for_completion
    -> dm_wait_for_bios_completion
     -> Unable to complete because of IO's on a plug in Task 2

  Task 2
  wb_workfn
  -> wb_writeback
   -> blk_start_plug
    -> writeback_sb_inodes
     -> Infinite loop unable to make an allocation

  Task 3
  cache_block_group
  ->read_extent_buffer_pages
   ->Waiting for IO to complete that can't be submitted because Task 1
     suspended the DM device

The problem here is that we need Task 2 to be scheduled completely for
the blk plug to flush.  Normally this would happen, we normally wait for
the block group caching to finish (Task 3), and this schedule would
result in the block plug flushing.

However if there's enough free space available from the current caching
to satisfy the allocation we won't actually wait for the caching to
complete.  This check however just checks that we have enough space, not
that we can make the allocation.  In this particular case we were trying
to allocate 9MiB, and we had 10MiB of free space, but we didn't have
9MiB of contiguous space to allocate, and thus the allocation failed and
we looped.

We specifically don't cycle through the FFE loop until we stop finding
cached block groups because we don't want to allocate new block groups
just because we're caching, so we short circuit the normal loop once we
hit LOOP_CACHING_WAIT and we found a caching block group.

This is normally fine, except in this particular case where the caching
thread can't make progress because the DM device has been suspended.

Fix this by not only waiting for free space to >= the amount of space we
want to allocate, but also that we make some progress in caching from
the time we start waiting.  This will keep us from busy looping when the
caching is taking a while but still theoretically has enough space for
us to allocate from, and fixes this particular case by forcing us to
actually sleep and wait for forward progress, which will flush the plug.

With this fix we're no longer hanging with generic/475.

CC: stable@vger.kernel.org # 6.1+
Reviewed-by: Boris Burkov <boris@bur.io>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: David Sterba <dsterba@suse.com>

diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index 63c3b7172ba54f6ba9773b8a511e6a3bac7d5790..1e4b70f5280d8ef9f3dc4dc8b534488721ce8771 100644 (file)
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -441,13 +441,23 @@ void btrfs_wait_block_group_cache_progress(struct btrfs_block_group *cache,
                                           u64 num_bytes)
 {
        struct btrfs_caching_control *caching_ctl;
+       int progress;
 
        caching_ctl = btrfs_get_caching_control(cache);
        if (!caching_ctl)
                return;
 
+       /*
+        * We've already failed to allocate from this block group, so even if
+        * there's enough space in the block group it isn't contiguous enough to
+        * allow for an allocation, so wait for at least the next wakeup tick,
+        * or for the thing to be done.
+        */
+       progress = atomic_read(&caching_ctl->progress);
+
        wait_event(caching_ctl->wait, btrfs_block_group_done(cache) ||
-                  (cache->free_space_ctl->free_space >= num_bytes));
+                  (progress != atomic_read(&caching_ctl->progress) &&
+                   (cache->free_space_ctl->free_space >= num_bytes)));
 
        btrfs_put_caching_control(caching_ctl);
 }
@@ -802,8 +812,10 @@ next:
 
                        if (total_found > CACHING_CTL_WAKE_UP) {
                                total_found = 0;
-                               if (wakeup)
+                               if (wakeup) {
+                                       atomic_inc(&caching_ctl->progress);
                                        wake_up(&caching_ctl->wait);
+                               }
                        }
                }
                path->slots[0]++;
@@ -910,6 +922,7 @@ int btrfs_cache_block_group(struct btrfs_block_group *cache, bool wait)
        init_waitqueue_head(&caching_ctl->wait);
        caching_ctl->block_group = cache;
        refcount_set(&caching_ctl->count, 2);
+       atomic_set(&caching_ctl->progress, 0);
        btrfs_init_work(&caching_ctl->work, caching_thread, NULL, NULL);
 
        spin_lock(&cache->lock);

diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index aba5dff66c19f1d2187be22423b193ff19d49c43..74b61e663028a866ee5fc3577f3f37fc8b245999 100644 (file)
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -90,6 +90,8 @@ struct btrfs_caching_control {
        wait_queue_head_t wait;
        struct btrfs_work work;
        struct btrfs_block_group *block_group;
+       /* Track progress of caching during allocation. */
+       atomic_t progress;
        refcount_t count;
 };