License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 22:07:57 +08:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2013-03-24 07:11:31 +08:00
|
|
|
/*
|
|
|
|
* bcache journalling code, for btree insertions
|
|
|
|
*
|
|
|
|
* Copyright 2012 Google, Inc.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include "bcache.h"
|
|
|
|
#include "btree.h"
|
|
|
|
#include "debug.h"
|
2014-04-11 08:58:49 +08:00
|
|
|
#include "extents.h"
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-04-27 06:39:55 +08:00
|
|
|
#include <trace/events/bcache.h>
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
/*
|
|
|
|
* Journal replay/recovery:
|
|
|
|
*
|
|
|
|
* This code is all driven from run_cache_set(); we first read the journal
|
|
|
|
* entries, do some other stuff, then we mark all the keys in the journal
|
|
|
|
* entries (same as garbage collection would), then we replay them - reinserting
|
|
|
|
* them into the cache in precisely the same order as they appear in the
|
|
|
|
* journal.
|
|
|
|
*
|
|
|
|
* We only journal keys that go in leaf nodes, which simplifies things quite a
|
|
|
|
* bit.
|
|
|
|
*/
|
|
|
|
|
2015-07-20 21:29:37 +08:00
|
|
|
static void journal_read_endio(struct bio *bio)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
|
|
|
struct closure *cl = bio->bi_private;
|
2018-08-11 13:19:45 +08:00
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
closure_put(cl);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int journal_read_bucket(struct cache *ca, struct list_head *list,
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int bucket_index)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
|
|
|
struct journal_device *ja = &ca->journal;
|
|
|
|
struct bio *bio = &ja->bio;
|
|
|
|
|
|
|
|
struct journal_replay *i;
|
|
|
|
struct jset *j, *data = ca->set->journal.w[0].data;
|
2013-07-25 08:44:17 +08:00
|
|
|
struct closure cl;
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int len, left, offset = 0;
|
2013-03-24 07:11:31 +08:00
|
|
|
int ret = 0;
|
|
|
|
sector_t bucket = bucket_to_sector(ca->set, ca->sb.d[bucket_index]);
|
|
|
|
|
2013-07-25 08:44:17 +08:00
|
|
|
closure_init_stack(&cl);
|
|
|
|
|
2013-08-06 05:04:06 +08:00
|
|
|
pr_debug("reading %u", bucket_index);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
while (offset < ca->sb.bucket_size) {
|
|
|
|
reread: left = ca->sb.bucket_size - offset;
|
2018-08-11 13:19:44 +08:00
|
|
|
len = min_t(unsigned int, left, PAGE_SECTORS << JSET_BITS);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
bio_reset(bio);
|
2013-10-12 06:44:27 +08:00
|
|
|
bio->bi_iter.bi_sector = bucket + offset;
|
2017-08-24 01:10:32 +08:00
|
|
|
bio_set_dev(bio, ca->bdev);
|
2013-10-12 06:44:27 +08:00
|
|
|
bio->bi_iter.bi_size = len << 9;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
bio->bi_end_io = journal_read_endio;
|
2013-07-25 08:44:17 +08:00
|
|
|
bio->bi_private = &cl;
|
2016-06-06 03:32:05 +08:00
|
|
|
bio_set_op_attrs(bio, REQ_OP_READ, 0);
|
2013-03-29 02:50:55 +08:00
|
|
|
bch_bio_map(bio, data);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
bcache: add CACHE_SET_IO_DISABLE to struct cache_set flags
When too many I/Os failed on cache device, bch_cache_set_error() is called
in the error handling code path to retire whole problematic cache set. If
new I/O requests continue to come and take refcount dc->count, the cache
set won't be retired immediately, this is a problem.
Further more, there are several kernel thread and self-armed kernel work
may still running after bch_cache_set_error() is called. It needs to wait
quite a while for them to stop, or they won't stop at all. They also
prevent the cache set from being retired.
The solution in this patch is, to add per cache set flag to disable I/O
request on this cache and all attached backing devices. Then new coming I/O
requests can be rejected in *_make_request() before taking refcount, kernel
threads and self-armed kernel worker can stop very fast when flags bit
CACHE_SET_IO_DISABLE is set.
Because bcache also do internal I/Os for writeback, garbage collection,
bucket allocation, journaling, this kind of I/O should be disabled after
bch_cache_set_error() is called. So closure_bio_submit() is modified to
check whether CACHE_SET_IO_DISABLE is set on cache_set->flags. If set,
closure_bio_submit() will set bio->bi_status to BLK_STS_IOERR and
return, generic_make_request() won't be called.
A sysfs interface is also added to set or clear CACHE_SET_IO_DISABLE bit
from cache_set->flags, to disable or enable cache set I/O for debugging. It
is helpful to trigger more corner case issues for failed cache device.
Changelog
v4, add wait_for_kthread_stop(), and call it before exits writeback and gc
kernel threads.
v3, change CACHE_SET_IO_DISABLE from 4 to 3, since it is bit index.
remove "bcache: " prefix when printing out kernel message.
v2, more changes by previous review,
- Use CACHE_SET_IO_DISABLE of cache_set->flags, suggested by Junhui.
- Check CACHE_SET_IO_DISABLE in bch_btree_gc() to stop a while-loop, this
is reported and inspired from origal patch of Pavel Vazharov.
v1, initial version.
Signed-off-by: Coly Li <colyli@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Reviewed-by: Michael Lyle <mlyle@lyle.org>
Cc: Junhui Tang <tang.junhui@zte.com.cn>
Cc: Michael Lyle <mlyle@lyle.org>
Cc: Pavel Vazharov <freakpv@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-19 08:36:17 +08:00
|
|
|
closure_bio_submit(ca->set, bio, &cl);
|
2013-07-25 08:44:17 +08:00
|
|
|
closure_sync(&cl);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
/* This function could be simpler now since we no longer write
|
|
|
|
* journal entries that overlap bucket boundaries; this means
|
|
|
|
* the start of a bucket will always have a valid journal entry
|
|
|
|
* if it has any journal entries at all.
|
|
|
|
*/
|
|
|
|
|
|
|
|
j = data;
|
|
|
|
while (len) {
|
|
|
|
struct list_head *where;
|
|
|
|
size_t blocks, bytes = set_bytes(j);
|
|
|
|
|
2013-08-06 05:04:06 +08:00
|
|
|
if (j->magic != jset_magic(&ca->sb)) {
|
|
|
|
pr_debug("%u: bad magic", bucket_index);
|
2013-03-24 07:11:31 +08:00
|
|
|
return ret;
|
2013-08-06 05:04:06 +08:00
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-08-06 05:04:06 +08:00
|
|
|
if (bytes > left << 9 ||
|
|
|
|
bytes > PAGE_SIZE << JSET_BITS) {
|
|
|
|
pr_info("%u: too big, %zu bytes, offset %u",
|
|
|
|
bucket_index, bytes, offset);
|
2013-03-24 07:11:31 +08:00
|
|
|
return ret;
|
2013-08-06 05:04:06 +08:00
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
if (bytes > len << 9)
|
|
|
|
goto reread;
|
|
|
|
|
2013-08-06 05:04:06 +08:00
|
|
|
if (j->csum != csum_set(j)) {
|
|
|
|
pr_info("%u: bad csum, %zu bytes, offset %u",
|
|
|
|
bucket_index, bytes, offset);
|
2013-03-24 07:11:31 +08:00
|
|
|
return ret;
|
2013-08-06 05:04:06 +08:00
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-12-18 15:49:49 +08:00
|
|
|
blocks = set_blocks(j, block_bytes(ca->set));
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2019-06-28 19:59:50 +08:00
|
|
|
/*
|
|
|
|
* Nodes in 'list' are in linear increasing order of
|
|
|
|
* i->j.seq, the node on head has the smallest (oldest)
|
|
|
|
* journal seq, the node on tail has the biggest
|
|
|
|
* (latest) journal seq.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check from the oldest jset for last_seq. If
|
|
|
|
* i->j.seq < j->last_seq, it means the oldest jset
|
|
|
|
* in list is expired and useless, remove it from
|
|
|
|
* this list. Otherwise, j is a condidate jset for
|
|
|
|
* further following checks.
|
|
|
|
*/
|
2013-03-24 07:11:31 +08:00
|
|
|
while (!list_empty(list)) {
|
|
|
|
i = list_first_entry(list,
|
|
|
|
struct journal_replay, list);
|
|
|
|
if (i->j.seq >= j->last_seq)
|
|
|
|
break;
|
|
|
|
list_del(&i->list);
|
|
|
|
kfree(i);
|
|
|
|
}
|
|
|
|
|
2019-06-28 19:59:50 +08:00
|
|
|
/* iterate list in reverse order (from latest jset) */
|
2013-03-24 07:11:31 +08:00
|
|
|
list_for_each_entry_reverse(i, list, list) {
|
|
|
|
if (j->seq == i->j.seq)
|
|
|
|
goto next_set;
|
|
|
|
|
2019-06-28 19:59:50 +08:00
|
|
|
/*
|
|
|
|
* if j->seq is less than any i->j.last_seq
|
|
|
|
* in list, j is an expired and useless jset.
|
|
|
|
*/
|
2013-03-24 07:11:31 +08:00
|
|
|
if (j->seq < i->j.last_seq)
|
|
|
|
goto next_set;
|
|
|
|
|
2019-06-28 19:59:50 +08:00
|
|
|
/*
|
|
|
|
* 'where' points to first jset in list which
|
|
|
|
* is elder then j.
|
|
|
|
*/
|
2013-03-24 07:11:31 +08:00
|
|
|
if (j->seq > i->j.seq) {
|
|
|
|
where = &i->list;
|
|
|
|
goto add;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
where = list;
|
|
|
|
add:
|
|
|
|
i = kmalloc(offsetof(struct journal_replay, j) +
|
|
|
|
bytes, GFP_KERNEL);
|
|
|
|
if (!i)
|
|
|
|
return -ENOMEM;
|
|
|
|
memcpy(&i->j, j, bytes);
|
2019-06-28 19:59:50 +08:00
|
|
|
/* Add to the location after 'where' points to */
|
2013-03-24 07:11:31 +08:00
|
|
|
list_add(&i->list, where);
|
|
|
|
ret = 1;
|
|
|
|
|
2019-06-28 19:59:51 +08:00
|
|
|
if (j->seq > ja->seq[bucket_index])
|
|
|
|
ja->seq[bucket_index] = j->seq;
|
2013-03-24 07:11:31 +08:00
|
|
|
next_set:
|
|
|
|
offset += blocks * ca->sb.block_size;
|
|
|
|
len -= blocks * ca->sb.block_size;
|
|
|
|
j = ((void *) j) + blocks * block_bytes(ca);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2013-07-25 08:44:17 +08:00
|
|
|
int bch_journal_read(struct cache_set *c, struct list_head *list)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
|
|
|
#define read_bucket(b) \
|
|
|
|
({ \
|
2019-04-25 00:48:32 +08:00
|
|
|
ret = journal_read_bucket(ca, list, b); \
|
2013-03-24 07:11:31 +08:00
|
|
|
__set_bit(b, bitmap); \
|
|
|
|
if (ret < 0) \
|
|
|
|
return ret; \
|
|
|
|
ret; \
|
|
|
|
})
|
|
|
|
|
|
|
|
struct cache *ca;
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int iter;
|
2019-04-25 00:48:32 +08:00
|
|
|
int ret = 0;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
for_each_cache(ca, c, iter) {
|
|
|
|
struct journal_device *ja = &ca->journal;
|
2015-07-01 05:59:54 +08:00
|
|
|
DECLARE_BITMAP(bitmap, SB_JOURNAL_BUCKETS);
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int i, l, r, m;
|
2013-03-24 07:11:31 +08:00
|
|
|
uint64_t seq;
|
|
|
|
|
|
|
|
bitmap_zero(bitmap, SB_JOURNAL_BUCKETS);
|
|
|
|
pr_debug("%u journal buckets", ca->sb.njournal_buckets);
|
|
|
|
|
2013-09-24 14:17:29 +08:00
|
|
|
/*
|
|
|
|
* Read journal buckets ordered by golden ratio hash to quickly
|
2013-03-24 07:11:31 +08:00
|
|
|
* find a sequence of buckets with valid journal entries
|
|
|
|
*/
|
|
|
|
for (i = 0; i < ca->sb.njournal_buckets; i++) {
|
bcache: add a comment in journal bucket reading
Journal bucket is a circular buffer, the bucket
can be like YYYNNNYY, which means the first valid journal in
the 7th bucket, and the latest valid journal in third bucket, in
this case, if we do not try we the zero index first, We
may get a valid journal in the 7th bucket, then we call
find_next_bit(bitmap,ca->sb.njournal_buckets, l + 1) to get the
first invalid bucket after the 7th bucket, because all these
buckets is valid, so no bit 1 in bitmap, thus find_next_bit()
function would return with ca->sb.njournal_buckets (8). So, after
that, bcache only read journal in 7th and 8the bucket,
the first to the third buckets are lost.
So, it is important to let developer know that, we need to try
the zero index at first in the hash-search, and avoid any breaks
in future's code modification.
[ML: Fixed whitespace & formatting & file permissions]
Signed-off-by: Tang Junhui <tang.junhui@zte.com.cn>
Signed-off-by: Michael Lyle <mlyle@lyle.org>
Reviewed-by: Michael Lyle <mlyle@lyle.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-11-25 07:14:24 +08:00
|
|
|
/*
|
|
|
|
* We must try the index l with ZERO first for
|
|
|
|
* correctness due to the scenario that the journal
|
|
|
|
* bucket is circular buffer which might have wrapped
|
|
|
|
*/
|
2013-03-24 07:11:31 +08:00
|
|
|
l = (i * 2654435769U) % ca->sb.njournal_buckets;
|
|
|
|
|
|
|
|
if (test_bit(l, bitmap))
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (read_bucket(l))
|
|
|
|
goto bsearch;
|
|
|
|
}
|
|
|
|
|
2013-09-24 14:17:29 +08:00
|
|
|
/*
|
|
|
|
* If that fails, check all the buckets we haven't checked
|
2013-03-24 07:11:31 +08:00
|
|
|
* already
|
|
|
|
*/
|
|
|
|
pr_debug("falling back to linear search");
|
|
|
|
|
2013-09-24 14:17:29 +08:00
|
|
|
for (l = find_first_zero_bit(bitmap, ca->sb.njournal_buckets);
|
|
|
|
l < ca->sb.njournal_buckets;
|
2018-08-11 13:19:47 +08:00
|
|
|
l = find_next_zero_bit(bitmap, ca->sb.njournal_buckets,
|
|
|
|
l + 1))
|
2013-03-24 07:11:31 +08:00
|
|
|
if (read_bucket(l))
|
|
|
|
goto bsearch;
|
2013-09-24 14:17:29 +08:00
|
|
|
|
2014-06-03 06:39:44 +08:00
|
|
|
/* no journal entries on this device? */
|
|
|
|
if (l == ca->sb.njournal_buckets)
|
2013-09-24 14:17:29 +08:00
|
|
|
continue;
|
2013-03-24 07:11:31 +08:00
|
|
|
bsearch:
|
2014-06-03 06:39:44 +08:00
|
|
|
BUG_ON(list_empty(list));
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
/* Binary search */
|
2014-04-11 07:09:51 +08:00
|
|
|
m = l;
|
|
|
|
r = find_next_bit(bitmap, ca->sb.njournal_buckets, l + 1);
|
2013-03-24 07:11:31 +08:00
|
|
|
pr_debug("starting binary search, l %u r %u", l, r);
|
|
|
|
|
|
|
|
while (l + 1 < r) {
|
2013-07-12 13:42:14 +08:00
|
|
|
seq = list_entry(list->prev, struct journal_replay,
|
|
|
|
list)->j.seq;
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
m = (l + r) >> 1;
|
2013-07-12 13:42:14 +08:00
|
|
|
read_bucket(m);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-07-12 13:42:14 +08:00
|
|
|
if (seq != list_entry(list->prev, struct journal_replay,
|
|
|
|
list)->j.seq)
|
2013-03-24 07:11:31 +08:00
|
|
|
l = m;
|
|
|
|
else
|
|
|
|
r = m;
|
|
|
|
}
|
|
|
|
|
2013-09-24 14:17:29 +08:00
|
|
|
/*
|
|
|
|
* Read buckets in reverse order until we stop finding more
|
2013-03-24 07:11:31 +08:00
|
|
|
* journal entries
|
|
|
|
*/
|
2013-09-24 14:17:29 +08:00
|
|
|
pr_debug("finishing up: m %u njournal_buckets %u",
|
|
|
|
m, ca->sb.njournal_buckets);
|
2013-03-24 07:11:31 +08:00
|
|
|
l = m;
|
|
|
|
|
|
|
|
while (1) {
|
|
|
|
if (!l--)
|
|
|
|
l = ca->sb.njournal_buckets - 1;
|
|
|
|
|
|
|
|
if (l == m)
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (test_bit(l, bitmap))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (!read_bucket(l))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
seq = 0;
|
|
|
|
|
|
|
|
for (i = 0; i < ca->sb.njournal_buckets; i++)
|
|
|
|
if (ja->seq[i] > seq) {
|
|
|
|
seq = ja->seq[i];
|
2014-03-14 04:44:21 +08:00
|
|
|
/*
|
|
|
|
* When journal_reclaim() goes to allocate for
|
|
|
|
* the first time, it'll use the bucket after
|
|
|
|
* ja->cur_idx
|
|
|
|
*/
|
|
|
|
ja->cur_idx = i;
|
|
|
|
ja->last_idx = ja->discard_idx = (i + 1) %
|
|
|
|
ca->sb.njournal_buckets;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-09-24 14:17:29 +08:00
|
|
|
if (!list_empty(list))
|
|
|
|
c->journal.seq = list_entry(list->prev,
|
|
|
|
struct journal_replay,
|
|
|
|
list)->j.seq;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2019-06-28 19:59:26 +08:00
|
|
|
return 0;
|
2013-03-24 07:11:31 +08:00
|
|
|
#undef read_bucket
|
|
|
|
}
|
|
|
|
|
|
|
|
void bch_journal_mark(struct cache_set *c, struct list_head *list)
|
|
|
|
{
|
|
|
|
atomic_t p = { 0 };
|
|
|
|
struct bkey *k;
|
|
|
|
struct journal_replay *i;
|
|
|
|
struct journal *j = &c->journal;
|
|
|
|
uint64_t last = j->seq;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* journal.pin should never fill up - we never write a journal
|
|
|
|
* entry when it would fill up. But if for some reason it does, we
|
|
|
|
* iterate over the list in reverse order so that we can just skip that
|
|
|
|
* refcount instead of bugging.
|
|
|
|
*/
|
|
|
|
|
|
|
|
list_for_each_entry_reverse(i, list, list) {
|
|
|
|
BUG_ON(last < i->j.seq);
|
|
|
|
i->pin = NULL;
|
|
|
|
|
|
|
|
while (last-- != i->j.seq)
|
|
|
|
if (fifo_free(&j->pin) > 1) {
|
|
|
|
fifo_push_front(&j->pin, p);
|
|
|
|
atomic_set(&fifo_front(&j->pin), 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (fifo_free(&j->pin) > 1) {
|
|
|
|
fifo_push_front(&j->pin, p);
|
|
|
|
i->pin = &fifo_front(&j->pin);
|
|
|
|
atomic_set(i->pin, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
for (k = i->j.start;
|
2013-12-18 13:56:21 +08:00
|
|
|
k < bset_bkey_last(&i->j);
|
2014-04-11 08:58:49 +08:00
|
|
|
k = bkey_next(k))
|
|
|
|
if (!__bch_extent_invalid(c, k)) {
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int j;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2014-04-11 08:58:49 +08:00
|
|
|
for (j = 0; j < KEY_PTRS(k); j++)
|
|
|
|
if (ptr_available(c, k, j))
|
|
|
|
atomic_inc(&PTR_BUCKET(c, k, j)->pin);
|
2014-02-25 11:55:28 +08:00
|
|
|
|
2014-04-11 08:58:49 +08:00
|
|
|
bch_initial_mark_key(c, 0, k);
|
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-05-01 20:34:09 +08:00
|
|
|
static bool is_discard_enabled(struct cache_set *s)
|
bcache: fix failure in journal relplay
journal replay failed with messages:
Sep 10 19:10:43 ceph kernel: bcache: error on
bb379a64-e44e-4812-b91d-a5599871a3b1: bcache: journal entries
2057493-2057567 missing! (replaying 2057493-2076601), disabling
caching
The reason is in journal_reclaim(), when discard is enabled, we send
discard command and reclaim those journal buckets whose seq is old
than the last_seq_now, but before we write a journal with last_seq_now,
the machine is restarted, so the journal with the last_seq_now is not
written to the journal bucket, and the last_seq_wrote in the newest
journal is old than last_seq_now which we expect to be, so when we doing
replay, journals from last_seq_wrote to last_seq_now are missing.
It's hard to write a journal immediately after journal_reclaim(),
and it harmless if those missed journal are caused by discarding
since those journals are already wrote to btree node. So, if miss
seqs are started from the beginning journal, we treat it as normal,
and only print a message to show the miss journal, and point out
it maybe caused by discarding.
Patch v2 add a judgement condition to ignore the missed journal
only when discard enabled as Coly suggested.
(Coly Li: rebase the patch with other changes in bch_journal_replay())
Signed-off-by: Tang Junhui <tang.junhui.linux@gmail.com>
Tested-by: Dennis Schridde <devurandom@gmx.net>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-25 00:48:41 +08:00
|
|
|
{
|
|
|
|
struct cache *ca;
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
for_each_cache(ca, s, i)
|
|
|
|
if (ca->discard)
|
|
|
|
return true;
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2013-07-25 08:44:17 +08:00
|
|
|
int bch_journal_replay(struct cache_set *s, struct list_head *list)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
|
|
|
int ret = 0, keys = 0, entries = 0;
|
|
|
|
struct bkey *k;
|
|
|
|
struct journal_replay *i =
|
|
|
|
list_entry(list->prev, struct journal_replay, list);
|
|
|
|
|
|
|
|
uint64_t start = i->j.last_seq, end = i->j.seq, n = start;
|
2013-07-25 08:26:51 +08:00
|
|
|
struct keylist keylist;
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
list_for_each_entry(i, list, list) {
|
|
|
|
BUG_ON(i->pin && atomic_read(i->pin) != 1);
|
|
|
|
|
2019-04-25 00:48:36 +08:00
|
|
|
if (n != i->j.seq) {
|
bcache: fix failure in journal relplay
journal replay failed with messages:
Sep 10 19:10:43 ceph kernel: bcache: error on
bb379a64-e44e-4812-b91d-a5599871a3b1: bcache: journal entries
2057493-2057567 missing! (replaying 2057493-2076601), disabling
caching
The reason is in journal_reclaim(), when discard is enabled, we send
discard command and reclaim those journal buckets whose seq is old
than the last_seq_now, but before we write a journal with last_seq_now,
the machine is restarted, so the journal with the last_seq_now is not
written to the journal bucket, and the last_seq_wrote in the newest
journal is old than last_seq_now which we expect to be, so when we doing
replay, journals from last_seq_wrote to last_seq_now are missing.
It's hard to write a journal immediately after journal_reclaim(),
and it harmless if those missed journal are caused by discarding
since those journals are already wrote to btree node. So, if miss
seqs are started from the beginning journal, we treat it as normal,
and only print a message to show the miss journal, and point out
it maybe caused by discarding.
Patch v2 add a judgement condition to ignore the missed journal
only when discard enabled as Coly suggested.
(Coly Li: rebase the patch with other changes in bch_journal_replay())
Signed-off-by: Tang Junhui <tang.junhui.linux@gmail.com>
Tested-by: Dennis Schridde <devurandom@gmx.net>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-25 00:48:41 +08:00
|
|
|
if (n == start && is_discard_enabled(s))
|
|
|
|
pr_info("bcache: journal entries %llu-%llu may be discarded! (replaying %llu-%llu)",
|
|
|
|
n, i->j.seq - 1, start, end);
|
|
|
|
else {
|
|
|
|
pr_err("bcache: journal entries %llu-%llu missing! (replaying %llu-%llu)",
|
|
|
|
n, i->j.seq - 1, start, end);
|
|
|
|
ret = -EIO;
|
|
|
|
goto err;
|
|
|
|
}
|
2019-04-25 00:48:36 +08:00
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
for (k = i->j.start;
|
2013-12-18 13:56:21 +08:00
|
|
|
k < bset_bkey_last(&i->j);
|
2013-03-24 07:11:31 +08:00
|
|
|
k = bkey_next(k)) {
|
2013-04-27 06:39:55 +08:00
|
|
|
trace_bcache_journal_replay_key(k);
|
|
|
|
|
2014-01-09 13:22:02 +08:00
|
|
|
bch_keylist_init_single(&keylist, k);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-07-25 09:07:22 +08:00
|
|
|
ret = bch_btree_insert(s, &keylist, i->pin, NULL);
|
2013-03-24 07:11:31 +08:00
|
|
|
if (ret)
|
|
|
|
goto err;
|
|
|
|
|
2013-07-25 08:26:51 +08:00
|
|
|
BUG_ON(!bch_keylist_empty(&keylist));
|
2013-03-24 07:11:31 +08:00
|
|
|
keys++;
|
|
|
|
|
|
|
|
cond_resched();
|
|
|
|
}
|
|
|
|
|
|
|
|
if (i->pin)
|
|
|
|
atomic_dec(i->pin);
|
|
|
|
n = i->j.seq + 1;
|
|
|
|
entries++;
|
|
|
|
}
|
|
|
|
|
|
|
|
pr_info("journal replay done, %i keys in %i entries, seq %llu",
|
|
|
|
keys, entries, end);
|
2013-07-25 09:04:18 +08:00
|
|
|
err:
|
2013-03-24 07:11:31 +08:00
|
|
|
while (!list_empty(list)) {
|
|
|
|
i = list_first_entry(list, struct journal_replay, list);
|
|
|
|
list_del(&i->list);
|
|
|
|
kfree(i);
|
|
|
|
}
|
2013-07-25 09:04:18 +08:00
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Journalling */
|
|
|
|
|
|
|
|
static void btree_flush_write(struct cache_set *c)
|
|
|
|
{
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
struct btree *b, *t, *btree_nodes[BTREE_FLUSH_NR];
|
2020-02-01 22:42:34 +08:00
|
|
|
unsigned int i, nr;
|
|
|
|
int ref_nr;
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
atomic_t *fifo_front_p, *now_fifo_front_p;
|
|
|
|
size_t mask;
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
|
|
|
|
if (c->journal.btree_flushing)
|
|
|
|
return;
|
|
|
|
|
|
|
|
spin_lock(&c->journal.flush_write_lock);
|
|
|
|
if (c->journal.btree_flushing) {
|
|
|
|
spin_unlock(&c->journal.flush_write_lock);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
c->journal.btree_flushing = true;
|
|
|
|
spin_unlock(&c->journal.flush_write_lock);
|
2018-02-08 03:41:39 +08:00
|
|
|
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
/* get the oldest journal entry and check its refcount */
|
|
|
|
spin_lock(&c->journal.lock);
|
|
|
|
fifo_front_p = &fifo_front(&c->journal.pin);
|
|
|
|
ref_nr = atomic_read(fifo_front_p);
|
|
|
|
if (ref_nr <= 0) {
|
|
|
|
/*
|
|
|
|
* do nothing if no btree node references
|
|
|
|
* the oldest journal entry
|
|
|
|
*/
|
|
|
|
spin_unlock(&c->journal.lock);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
spin_unlock(&c->journal.lock);
|
|
|
|
|
|
|
|
mask = c->journal.pin.mask;
|
|
|
|
nr = 0;
|
2018-02-08 03:41:39 +08:00
|
|
|
atomic_long_inc(&c->flush_write);
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
memset(btree_nodes, 0, sizeof(btree_nodes));
|
2019-06-28 19:59:54 +08:00
|
|
|
|
bcache: fix race in btree_flush_write()
There is a race between mca_reap(), btree_node_free() and journal code
btree_flush_write(), which results very rare and strange deadlock or
panic and are very hard to reproduce.
Let me explain how the race happens. In btree_flush_write() one btree
node with oldest journal pin is selected, then it is flushed to cache
device, the select-and-flush is a two steps operation. Between these two
steps, there are something may happen inside the race window,
- The selected btree node was reaped by mca_reap() and allocated to
other requesters for other btree node.
- The slected btree node was selected, flushed and released by mca
shrink callback bch_mca_scan().
When btree_flush_write() tries to flush the selected btree node, firstly
b->write_lock is held by mutex_lock(). If the race happens and the
memory of selected btree node is allocated to other btree node, if that
btree node's write_lock is held already, a deadlock very probably
happens here. A worse case is the memory of the selected btree node is
released, then all references to this btree node (e.g. b->write_lock)
will trigger NULL pointer deference panic.
This race was introduced in commit cafe56359144 ("bcache: A block layer
cache"), and enlarged by commit c4dc2497d50d ("bcache: fix high CPU
occupancy during journal"), which selected 128 btree nodes and flushed
them one-by-one in a quite long time period.
Such race is not easy to reproduce before. On a Lenovo SR650 server with
48 Xeon cores, and configure 1 NVMe SSD as cache device, a MD raid0
device assembled by 3 NVMe SSDs as backing device, this race can be
observed around every 10,000 times btree_flush_write() gets called. Both
deadlock and kernel panic all happened as aftermath of the race.
The idea of the fix is to add a btree flag BTREE_NODE_journal_flush. It
is set when selecting btree nodes, and cleared after btree nodes
flushed. Then when mca_reap() selects a btree node with this bit set,
this btree node will be skipped. Since mca_reap() only reaps btree node
without BTREE_NODE_journal_flush flag, such race is avoided.
Once corner case should be noticed, that is btree_node_free(). It might
be called in some error handling code path. For example the following
code piece from btree_split(),
2149 err_free2:
2150 bkey_put(b->c, &n2->key);
2151 btree_node_free(n2);
2152 rw_unlock(true, n2);
2153 err_free1:
2154 bkey_put(b->c, &n1->key);
2155 btree_node_free(n1);
2156 rw_unlock(true, n1);
At line 2151 and 2155, the btree node n2 and n1 are released without
mac_reap(), so BTREE_NODE_journal_flush also needs to be checked here.
If btree_node_free() is called directly in such error handling path,
and the selected btree node has BTREE_NODE_journal_flush bit set, just
delay for 1 us and retry again. In this case this btree node won't
be skipped, just retry until the BTREE_NODE_journal_flush bit cleared,
and free the btree node memory.
Fixes: cafe56359144 ("bcache: A block layer cache")
Signed-off-by: Coly Li <colyli@suse.de>
Reported-and-tested-by: kbuild test robot <lkp@intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:58 +08:00
|
|
|
mutex_lock(&c->bucket_lock);
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
list_for_each_entry_safe_reverse(b, t, &c->btree_cache, list) {
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
/*
|
|
|
|
* It is safe to get now_fifo_front_p without holding
|
|
|
|
* c->journal.lock here, because we don't need to know
|
|
|
|
* the exactly accurate value, just check whether the
|
|
|
|
* front pointer of c->journal.pin is changed.
|
|
|
|
*/
|
|
|
|
now_fifo_front_p = &fifo_front(&c->journal.pin);
|
|
|
|
/*
|
|
|
|
* If the oldest journal entry is reclaimed and front
|
|
|
|
* pointer of c->journal.pin changes, it is unnecessary
|
|
|
|
* to scan c->btree_cache anymore, just quit the loop and
|
|
|
|
* flush out what we have already.
|
|
|
|
*/
|
|
|
|
if (now_fifo_front_p != fifo_front_p)
|
|
|
|
break;
|
|
|
|
/*
|
|
|
|
* quit this loop if all matching btree nodes are
|
|
|
|
* scanned and record in btree_nodes[] already.
|
|
|
|
*/
|
|
|
|
ref_nr = atomic_read(fifo_front_p);
|
|
|
|
if (nr >= ref_nr)
|
|
|
|
break;
|
|
|
|
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
if (btree_node_journal_flush(b))
|
|
|
|
pr_err("BUG: flush_write bit should not be set here!");
|
|
|
|
|
|
|
|
mutex_lock(&b->write_lock);
|
|
|
|
|
|
|
|
if (!btree_node_dirty(b)) {
|
|
|
|
mutex_unlock(&b->write_lock);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!btree_current_write(b)->journal) {
|
|
|
|
mutex_unlock(&b->write_lock);
|
|
|
|
continue;
|
2019-06-28 19:59:54 +08:00
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
/*
|
|
|
|
* Only select the btree node which exactly references
|
|
|
|
* the oldest journal entry.
|
|
|
|
*
|
|
|
|
* If the journal entry pointed by fifo_front_p is
|
|
|
|
* reclaimed in parallel, don't worry:
|
|
|
|
* - the list_for_each_xxx loop will quit when checking
|
|
|
|
* next now_fifo_front_p.
|
|
|
|
* - If there are matched nodes recorded in btree_nodes[],
|
|
|
|
* they are clean now (this is why and how the oldest
|
|
|
|
* journal entry can be reclaimed). These selected nodes
|
|
|
|
* will be ignored and skipped in the folowing for-loop.
|
|
|
|
*/
|
2020-02-13 22:12:07 +08:00
|
|
|
if (((btree_current_write(b)->journal - fifo_front_p) &
|
|
|
|
mask) != 0) {
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
mutex_unlock(&b->write_lock);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
bcache: fix race in btree_flush_write()
There is a race between mca_reap(), btree_node_free() and journal code
btree_flush_write(), which results very rare and strange deadlock or
panic and are very hard to reproduce.
Let me explain how the race happens. In btree_flush_write() one btree
node with oldest journal pin is selected, then it is flushed to cache
device, the select-and-flush is a two steps operation. Between these two
steps, there are something may happen inside the race window,
- The selected btree node was reaped by mca_reap() and allocated to
other requesters for other btree node.
- The slected btree node was selected, flushed and released by mca
shrink callback bch_mca_scan().
When btree_flush_write() tries to flush the selected btree node, firstly
b->write_lock is held by mutex_lock(). If the race happens and the
memory of selected btree node is allocated to other btree node, if that
btree node's write_lock is held already, a deadlock very probably
happens here. A worse case is the memory of the selected btree node is
released, then all references to this btree node (e.g. b->write_lock)
will trigger NULL pointer deference panic.
This race was introduced in commit cafe56359144 ("bcache: A block layer
cache"), and enlarged by commit c4dc2497d50d ("bcache: fix high CPU
occupancy during journal"), which selected 128 btree nodes and flushed
them one-by-one in a quite long time period.
Such race is not easy to reproduce before. On a Lenovo SR650 server with
48 Xeon cores, and configure 1 NVMe SSD as cache device, a MD raid0
device assembled by 3 NVMe SSDs as backing device, this race can be
observed around every 10,000 times btree_flush_write() gets called. Both
deadlock and kernel panic all happened as aftermath of the race.
The idea of the fix is to add a btree flag BTREE_NODE_journal_flush. It
is set when selecting btree nodes, and cleared after btree nodes
flushed. Then when mca_reap() selects a btree node with this bit set,
this btree node will be skipped. Since mca_reap() only reaps btree node
without BTREE_NODE_journal_flush flag, such race is avoided.
Once corner case should be noticed, that is btree_node_free(). It might
be called in some error handling code path. For example the following
code piece from btree_split(),
2149 err_free2:
2150 bkey_put(b->c, &n2->key);
2151 btree_node_free(n2);
2152 rw_unlock(true, n2);
2153 err_free1:
2154 bkey_put(b->c, &n1->key);
2155 btree_node_free(n1);
2156 rw_unlock(true, n1);
At line 2151 and 2155, the btree node n2 and n1 are released without
mac_reap(), so BTREE_NODE_journal_flush also needs to be checked here.
If btree_node_free() is called directly in such error handling path,
and the selected btree node has BTREE_NODE_journal_flush bit set, just
delay for 1 us and retry again. In this case this btree node won't
be skipped, just retry until the BTREE_NODE_journal_flush bit cleared,
and free the btree node memory.
Fixes: cafe56359144 ("bcache: A block layer cache")
Signed-off-by: Coly Li <colyli@suse.de>
Reported-and-tested-by: kbuild test robot <lkp@intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:58 +08:00
|
|
|
set_btree_node_journal_flush(b);
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
|
|
|
|
mutex_unlock(&b->write_lock);
|
|
|
|
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
btree_nodes[nr++] = b;
|
|
|
|
/*
|
|
|
|
* To avoid holding c->bucket_lock too long time,
|
|
|
|
* only scan for BTREE_FLUSH_NR matched btree nodes
|
|
|
|
* at most. If there are more btree nodes reference
|
|
|
|
* the oldest journal entry, try to flush them next
|
|
|
|
* time when btree_flush_write() is called.
|
|
|
|
*/
|
|
|
|
if (nr == BTREE_FLUSH_NR)
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
break;
|
|
|
|
}
|
bcache: fix race in btree_flush_write()
There is a race between mca_reap(), btree_node_free() and journal code
btree_flush_write(), which results very rare and strange deadlock or
panic and are very hard to reproduce.
Let me explain how the race happens. In btree_flush_write() one btree
node with oldest journal pin is selected, then it is flushed to cache
device, the select-and-flush is a two steps operation. Between these two
steps, there are something may happen inside the race window,
- The selected btree node was reaped by mca_reap() and allocated to
other requesters for other btree node.
- The slected btree node was selected, flushed and released by mca
shrink callback bch_mca_scan().
When btree_flush_write() tries to flush the selected btree node, firstly
b->write_lock is held by mutex_lock(). If the race happens and the
memory of selected btree node is allocated to other btree node, if that
btree node's write_lock is held already, a deadlock very probably
happens here. A worse case is the memory of the selected btree node is
released, then all references to this btree node (e.g. b->write_lock)
will trigger NULL pointer deference panic.
This race was introduced in commit cafe56359144 ("bcache: A block layer
cache"), and enlarged by commit c4dc2497d50d ("bcache: fix high CPU
occupancy during journal"), which selected 128 btree nodes and flushed
them one-by-one in a quite long time period.
Such race is not easy to reproduce before. On a Lenovo SR650 server with
48 Xeon cores, and configure 1 NVMe SSD as cache device, a MD raid0
device assembled by 3 NVMe SSDs as backing device, this race can be
observed around every 10,000 times btree_flush_write() gets called. Both
deadlock and kernel panic all happened as aftermath of the race.
The idea of the fix is to add a btree flag BTREE_NODE_journal_flush. It
is set when selecting btree nodes, and cleared after btree nodes
flushed. Then when mca_reap() selects a btree node with this bit set,
this btree node will be skipped. Since mca_reap() only reaps btree node
without BTREE_NODE_journal_flush flag, such race is avoided.
Once corner case should be noticed, that is btree_node_free(). It might
be called in some error handling code path. For example the following
code piece from btree_split(),
2149 err_free2:
2150 bkey_put(b->c, &n2->key);
2151 btree_node_free(n2);
2152 rw_unlock(true, n2);
2153 err_free1:
2154 bkey_put(b->c, &n1->key);
2155 btree_node_free(n1);
2156 rw_unlock(true, n1);
At line 2151 and 2155, the btree node n2 and n1 are released without
mac_reap(), so BTREE_NODE_journal_flush also needs to be checked here.
If btree_node_free() is called directly in such error handling path,
and the selected btree node has BTREE_NODE_journal_flush bit set, just
delay for 1 us and retry again. In this case this btree node won't
be skipped, just retry until the BTREE_NODE_journal_flush bit cleared,
and free the btree node memory.
Fixes: cafe56359144 ("bcache: A block layer cache")
Signed-off-by: Coly Li <colyli@suse.de>
Reported-and-tested-by: kbuild test robot <lkp@intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:58 +08:00
|
|
|
mutex_unlock(&c->bucket_lock);
|
|
|
|
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
for (i = 0; i < nr; i++) {
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
b = btree_nodes[i];
|
|
|
|
if (!b) {
|
|
|
|
pr_err("BUG: btree_nodes[%d] is NULL", i);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* safe to check without holding b->write_lock */
|
|
|
|
if (!btree_node_journal_flush(b)) {
|
|
|
|
pr_err("BUG: bnode %p: journal_flush bit cleaned", b);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2014-03-05 08:42:42 +08:00
|
|
|
mutex_lock(&b->write_lock);
|
2013-10-25 08:07:04 +08:00
|
|
|
if (!btree_current_write(b)->journal) {
|
bcache: fix race in btree_flush_write()
There is a race between mca_reap(), btree_node_free() and journal code
btree_flush_write(), which results very rare and strange deadlock or
panic and are very hard to reproduce.
Let me explain how the race happens. In btree_flush_write() one btree
node with oldest journal pin is selected, then it is flushed to cache
device, the select-and-flush is a two steps operation. Between these two
steps, there are something may happen inside the race window,
- The selected btree node was reaped by mca_reap() and allocated to
other requesters for other btree node.
- The slected btree node was selected, flushed and released by mca
shrink callback bch_mca_scan().
When btree_flush_write() tries to flush the selected btree node, firstly
b->write_lock is held by mutex_lock(). If the race happens and the
memory of selected btree node is allocated to other btree node, if that
btree node's write_lock is held already, a deadlock very probably
happens here. A worse case is the memory of the selected btree node is
released, then all references to this btree node (e.g. b->write_lock)
will trigger NULL pointer deference panic.
This race was introduced in commit cafe56359144 ("bcache: A block layer
cache"), and enlarged by commit c4dc2497d50d ("bcache: fix high CPU
occupancy during journal"), which selected 128 btree nodes and flushed
them one-by-one in a quite long time period.
Such race is not easy to reproduce before. On a Lenovo SR650 server with
48 Xeon cores, and configure 1 NVMe SSD as cache device, a MD raid0
device assembled by 3 NVMe SSDs as backing device, this race can be
observed around every 10,000 times btree_flush_write() gets called. Both
deadlock and kernel panic all happened as aftermath of the race.
The idea of the fix is to add a btree flag BTREE_NODE_journal_flush. It
is set when selecting btree nodes, and cleared after btree nodes
flushed. Then when mca_reap() selects a btree node with this bit set,
this btree node will be skipped. Since mca_reap() only reaps btree node
without BTREE_NODE_journal_flush flag, such race is avoided.
Once corner case should be noticed, that is btree_node_free(). It might
be called in some error handling code path. For example the following
code piece from btree_split(),
2149 err_free2:
2150 bkey_put(b->c, &n2->key);
2151 btree_node_free(n2);
2152 rw_unlock(true, n2);
2153 err_free1:
2154 bkey_put(b->c, &n1->key);
2155 btree_node_free(n1);
2156 rw_unlock(true, n1);
At line 2151 and 2155, the btree node n2 and n1 are released without
mac_reap(), so BTREE_NODE_journal_flush also needs to be checked here.
If btree_node_free() is called directly in such error handling path,
and the selected btree node has BTREE_NODE_journal_flush bit set, just
delay for 1 us and retry again. In this case this btree node won't
be skipped, just retry until the BTREE_NODE_journal_flush bit cleared,
and free the btree node memory.
Fixes: cafe56359144 ("bcache: A block layer cache")
Signed-off-by: Coly Li <colyli@suse.de>
Reported-and-tested-by: kbuild test robot <lkp@intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:58 +08:00
|
|
|
clear_bit(BTREE_NODE_journal_flush, &b->flags);
|
2014-03-05 08:42:42 +08:00
|
|
|
mutex_unlock(&b->write_lock);
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
pr_debug("bnode %p: written by others", b);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!btree_node_dirty(b)) {
|
|
|
|
clear_bit(BTREE_NODE_journal_flush, &b->flags);
|
|
|
|
mutex_unlock(&b->write_lock);
|
|
|
|
pr_debug("bnode %p: dirty bit cleaned by others", b);
|
|
|
|
continue;
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
2014-03-05 08:42:42 +08:00
|
|
|
__bch_btree_node_write(b, NULL);
|
bcache: fix race in btree_flush_write()
There is a race between mca_reap(), btree_node_free() and journal code
btree_flush_write(), which results very rare and strange deadlock or
panic and are very hard to reproduce.
Let me explain how the race happens. In btree_flush_write() one btree
node with oldest journal pin is selected, then it is flushed to cache
device, the select-and-flush is a two steps operation. Between these two
steps, there are something may happen inside the race window,
- The selected btree node was reaped by mca_reap() and allocated to
other requesters for other btree node.
- The slected btree node was selected, flushed and released by mca
shrink callback bch_mca_scan().
When btree_flush_write() tries to flush the selected btree node, firstly
b->write_lock is held by mutex_lock(). If the race happens and the
memory of selected btree node is allocated to other btree node, if that
btree node's write_lock is held already, a deadlock very probably
happens here. A worse case is the memory of the selected btree node is
released, then all references to this btree node (e.g. b->write_lock)
will trigger NULL pointer deference panic.
This race was introduced in commit cafe56359144 ("bcache: A block layer
cache"), and enlarged by commit c4dc2497d50d ("bcache: fix high CPU
occupancy during journal"), which selected 128 btree nodes and flushed
them one-by-one in a quite long time period.
Such race is not easy to reproduce before. On a Lenovo SR650 server with
48 Xeon cores, and configure 1 NVMe SSD as cache device, a MD raid0
device assembled by 3 NVMe SSDs as backing device, this race can be
observed around every 10,000 times btree_flush_write() gets called. Both
deadlock and kernel panic all happened as aftermath of the race.
The idea of the fix is to add a btree flag BTREE_NODE_journal_flush. It
is set when selecting btree nodes, and cleared after btree nodes
flushed. Then when mca_reap() selects a btree node with this bit set,
this btree node will be skipped. Since mca_reap() only reaps btree node
without BTREE_NODE_journal_flush flag, such race is avoided.
Once corner case should be noticed, that is btree_node_free(). It might
be called in some error handling code path. For example the following
code piece from btree_split(),
2149 err_free2:
2150 bkey_put(b->c, &n2->key);
2151 btree_node_free(n2);
2152 rw_unlock(true, n2);
2153 err_free1:
2154 bkey_put(b->c, &n1->key);
2155 btree_node_free(n1);
2156 rw_unlock(true, n1);
At line 2151 and 2155, the btree node n2 and n1 are released without
mac_reap(), so BTREE_NODE_journal_flush also needs to be checked here.
If btree_node_free() is called directly in such error handling path,
and the selected btree node has BTREE_NODE_journal_flush bit set, just
delay for 1 us and retry again. In this case this btree node won't
be skipped, just retry until the BTREE_NODE_journal_flush bit cleared,
and free the btree node memory.
Fixes: cafe56359144 ("bcache: A block layer cache")
Signed-off-by: Coly Li <colyli@suse.de>
Reported-and-tested-by: kbuild test robot <lkp@intel.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:58 +08:00
|
|
|
clear_bit(BTREE_NODE_journal_flush, &b->flags);
|
2014-03-05 08:42:42 +08:00
|
|
|
mutex_unlock(&b->write_lock);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
|
bcache: avoid unnecessary btree nodes flushing in btree_flush_write()
the commit 91be66e1318f ("bcache: performance improvement for
btree_flush_write()") was an effort to flushing btree node with oldest
btree node faster in following methods,
- Only iterate dirty btree nodes in c->btree_cache, avoid scanning a lot
of clean btree nodes.
- Take c->btree_cache as a LRU-like list, aggressively flushing all
dirty nodes from tail of c->btree_cache util the btree node with
oldest journal entry is flushed. This is to reduce the time of holding
c->bucket_lock.
Guoju Fang and Shuang Li reported that they observe unexptected extra
write I/Os on cache device after applying the above patch. Guoju Fang
provideed more detailed diagnose information that the aggressive
btree nodes flushing may cause 10x more btree nodes to flush in his
workload. He points out when system memory is large enough to hold all
btree nodes in memory, c->btree_cache is not a LRU-like list any more.
Then the btree node with oldest journal entry is very probably not-
close to the tail of c->btree_cache list. In such situation much more
dirty btree nodes will be aggressively flushed before the target node
is flushed. When slow SATA SSD is used as cache device, such over-
aggressive flushing behavior will cause performance regression.
After spending a lot of time on debug and diagnose, I find the real
condition is more complicated, aggressive flushing dirty btree nodes
from tail of c->btree_cache list is not a good solution.
- When all btree nodes are cached in memory, c->btree_cache is not
a LRU-like list, the btree nodes with oldest journal entry won't
be close to the tail of the list.
- There can be hundreds dirty btree nodes reference the oldest journal
entry, before flushing all the nodes the oldest journal entry cannot
be reclaimed.
When the above two conditions mixed together, a simply flushing from
tail of c->btree_cache list is really NOT a good idea.
Fortunately there is still chance to make btree_flush_write() work
better. Here is how this patch avoids unnecessary btree nodes flushing,
- Only acquire c->journal.lock when getting oldest journal entry of
fifo c->journal.pin. In rested locations check the journal entries
locklessly, so their values can be changed on other cores
in parallel.
- In loop list_for_each_entry_safe_reverse(), checking latest front
point of fifo c->journal.pin. If it is different from the original
point which we get with locking c->journal.lock, it means the oldest
journal entry is reclaim on other cores. At this moment, all selected
dirty nodes recorded in array btree_nodes[] are all flushed and clean
on other CPU cores, it is unncessary to iterate c->btree_cache any
longer. Just quit the list_for_each_entry_safe_reverse() loop and
the following for-loop will skip all the selected clean nodes.
- Find a proper time to quit the list_for_each_entry_safe_reverse()
loop. Check the refcount value of orignial fifo front point, if the
value is larger than selected node number of btree_nodes[], it means
more matching btree nodes should be scanned. Otherwise it means no
more matching btee nodes in rest of c->btree_cache list, the loop
can be quit. If the original oldest journal entry is reclaimed and
fifo front point is updated, the refcount of original fifo front point
will be 0, then the loop will be quit too.
- Not hold c->bucket_lock too long time. c->bucket_lock is also required
for space allocation for cached data, hold it for too long time will
block regular I/O requests. When iterating list c->btree_cache, even
there are a lot of maching btree nodes, in order to not holding
c->bucket_lock for too long time, only BTREE_FLUSH_NR nodes are
selected and to flush in following for-loop.
With this patch, only btree nodes referencing oldest journal entry
are flushed to cache device, no aggressive flushing for unnecessary
btree node any more. And in order to avoid blocking regluar I/O
requests, each time when btree_flush_write() called, at most only
BTREE_FLUSH_NR btree nodes are selected to flush, even there are more
maching btree nodes in list c->btree_cache.
At last, one more thing to explain: Why it is safe to read front point
of c->journal.pin without holding c->journal.lock inside the
list_for_each_entry_safe_reverse() loop ?
Here is my answer: When reading the front point of fifo c->journal.pin,
we don't need to know the exact value of front point, we just want to
check whether the value is different from the original front point
(which is accurate value because we get it while c->jouranl.lock is
held). For such purpose, it works as expected without holding
c->journal.lock. Even the front point is changed on other CPU core and
not updated to local core, and current iterating btree node has
identical journal entry local as original fetched fifo front point, it
is still safe. Because after holding mutex b->write_lock (with memory
barrier) this btree node can be found as clean and skipped, the loop
will quite latter when iterate on next node of list c->btree_cache.
Fixes: 91be66e1318f ("bcache: performance improvement for btree_flush_write()")
Reported-by: Guoju Fang <fangguoju@gmail.com>
Reported-by: Shuang Li <psymon@bonuscloud.io>
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-01-24 01:01:37 +08:00
|
|
|
out:
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
spin_lock(&c->journal.flush_write_lock);
|
|
|
|
c->journal.btree_flushing = false;
|
|
|
|
spin_unlock(&c->journal.flush_write_lock);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
#define last_seq(j) ((j)->seq - fifo_used(&(j)->pin) + 1)
|
|
|
|
|
2015-07-20 21:29:37 +08:00
|
|
|
static void journal_discard_endio(struct bio *bio)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
|
|
|
struct journal_device *ja =
|
|
|
|
container_of(bio, struct journal_device, discard_bio);
|
|
|
|
struct cache *ca = container_of(ja, struct cache, journal);
|
|
|
|
|
|
|
|
atomic_set(&ja->discard_in_flight, DISCARD_DONE);
|
|
|
|
|
|
|
|
closure_wake_up(&ca->set->journal.wait);
|
|
|
|
closure_put(&ca->set->cl);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void journal_discard_work(struct work_struct *work)
|
|
|
|
{
|
|
|
|
struct journal_device *ja =
|
|
|
|
container_of(work, struct journal_device, discard_work);
|
|
|
|
|
2016-06-06 03:31:41 +08:00
|
|
|
submit_bio(&ja->discard_bio);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static void do_journal_discard(struct cache *ca)
|
|
|
|
{
|
|
|
|
struct journal_device *ja = &ca->journal;
|
|
|
|
struct bio *bio = &ja->discard_bio;
|
|
|
|
|
|
|
|
if (!ca->discard) {
|
|
|
|
ja->discard_idx = ja->last_idx;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2013-09-24 14:17:27 +08:00
|
|
|
switch (atomic_read(&ja->discard_in_flight)) {
|
2013-03-24 07:11:31 +08:00
|
|
|
case DISCARD_IN_FLIGHT:
|
|
|
|
return;
|
|
|
|
|
|
|
|
case DISCARD_DONE:
|
|
|
|
ja->discard_idx = (ja->discard_idx + 1) %
|
|
|
|
ca->sb.njournal_buckets;
|
|
|
|
|
|
|
|
atomic_set(&ja->discard_in_flight, DISCARD_READY);
|
|
|
|
/* fallthrough */
|
|
|
|
|
|
|
|
case DISCARD_READY:
|
|
|
|
if (ja->discard_idx == ja->last_idx)
|
|
|
|
return;
|
|
|
|
|
|
|
|
atomic_set(&ja->discard_in_flight, DISCARD_IN_FLIGHT);
|
|
|
|
|
2016-11-22 23:57:21 +08:00
|
|
|
bio_init(bio, bio->bi_inline_vecs, 1);
|
2016-06-06 03:32:05 +08:00
|
|
|
bio_set_op_attrs(bio, REQ_OP_DISCARD, 0);
|
2013-10-12 06:44:27 +08:00
|
|
|
bio->bi_iter.bi_sector = bucket_to_sector(ca->set,
|
2013-03-26 02:46:44 +08:00
|
|
|
ca->sb.d[ja->discard_idx]);
|
2017-08-24 01:10:32 +08:00
|
|
|
bio_set_dev(bio, ca->bdev);
|
2013-10-12 06:44:27 +08:00
|
|
|
bio->bi_iter.bi_size = bucket_bytes(ca);
|
2013-03-24 07:11:31 +08:00
|
|
|
bio->bi_end_io = journal_discard_endio;
|
|
|
|
|
|
|
|
closure_get(&ca->set->cl);
|
|
|
|
INIT_WORK(&ja->discard_work, journal_discard_work);
|
2018-09-27 23:41:46 +08:00
|
|
|
queue_work(bch_journal_wq, &ja->discard_work);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void journal_reclaim(struct cache_set *c)
|
|
|
|
{
|
|
|
|
struct bkey *k = &c->journal.key;
|
|
|
|
struct cache *ca;
|
|
|
|
uint64_t last_seq;
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int iter, n = 0;
|
2018-03-19 08:36:31 +08:00
|
|
|
atomic_t p __maybe_unused;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2018-02-08 03:41:39 +08:00
|
|
|
atomic_long_inc(&c->reclaim);
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
while (!atomic_read(&fifo_front(&c->journal.pin)))
|
|
|
|
fifo_pop(&c->journal.pin, p);
|
|
|
|
|
|
|
|
last_seq = last_seq(&c->journal);
|
|
|
|
|
|
|
|
/* Update last_idx */
|
|
|
|
|
|
|
|
for_each_cache(ca, c, iter) {
|
|
|
|
struct journal_device *ja = &ca->journal;
|
|
|
|
|
|
|
|
while (ja->last_idx != ja->cur_idx &&
|
|
|
|
ja->seq[ja->last_idx] < last_seq)
|
|
|
|
ja->last_idx = (ja->last_idx + 1) %
|
|
|
|
ca->sb.njournal_buckets;
|
|
|
|
}
|
|
|
|
|
|
|
|
for_each_cache(ca, c, iter)
|
|
|
|
do_journal_discard(ca);
|
|
|
|
|
|
|
|
if (c->journal.blocks_free)
|
2013-10-25 08:07:04 +08:00
|
|
|
goto out;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Allocate:
|
|
|
|
* XXX: Sort by free journal space
|
|
|
|
*/
|
|
|
|
|
|
|
|
for_each_cache(ca, c, iter) {
|
|
|
|
struct journal_device *ja = &ca->journal;
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int next = (ja->cur_idx + 1) % ca->sb.njournal_buckets;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
/* No space available on this device */
|
|
|
|
if (next == ja->discard_idx)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
ja->cur_idx = next;
|
2017-11-25 07:14:25 +08:00
|
|
|
k->ptr[n++] = MAKE_PTR(0,
|
2013-03-24 07:11:31 +08:00
|
|
|
bucket_to_sector(c, ca->sb.d[ja->cur_idx]),
|
|
|
|
ca->sb.nr_this_dev);
|
2019-06-28 20:00:00 +08:00
|
|
|
atomic_long_inc(&c->reclaimed_journal_buckets);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
bcache: never set KEY_PTRS of journal key to 0 in journal_reclaim()
In journal_reclaim() ja->cur_idx of each cache will be update to
reclaim available journal buckets. Variable 'int n' is used to count how
many cache is successfully reclaimed, then n is set to c->journal.key
by SET_KEY_PTRS(). Later in journal_write_unlocked(), a for_each_cache()
loop will write the jset data onto each cache.
The problem is, if all jouranl buckets on each cache is full, the
following code in journal_reclaim(),
529 for_each_cache(ca, c, iter) {
530 struct journal_device *ja = &ca->journal;
531 unsigned int next = (ja->cur_idx + 1) % ca->sb.njournal_buckets;
532
533 /* No space available on this device */
534 if (next == ja->discard_idx)
535 continue;
536
537 ja->cur_idx = next;
538 k->ptr[n++] = MAKE_PTR(0,
539 bucket_to_sector(c, ca->sb.d[ja->cur_idx]),
540 ca->sb.nr_this_dev);
541 }
542
543 bkey_init(k);
544 SET_KEY_PTRS(k, n);
If there is no available bucket to reclaim, the if() condition at line
534 will always true, and n remains 0. Then at line 544, SET_KEY_PTRS()
will set KEY_PTRS field of c->journal.key to 0.
Setting KEY_PTRS field of c->journal.key to 0 is wrong. Because in
journal_write_unlocked() the journal data is written in following loop,
649 for (i = 0; i < KEY_PTRS(k); i++) {
650-671 submit journal data to cache device
672 }
If KEY_PTRS field is set to 0 in jouranl_reclaim(), the journal data
won't be written to cache device here. If system crahed or rebooted
before bkeys of the lost journal entries written into btree nodes, data
corruption will be reported during bcache reload after rebooting the
system.
Indeed there is only one cache in a cache set, there is no need to set
KEY_PTRS field in journal_reclaim() at all. But in order to keep the
for_each_cache() logic consistent for now, this patch fixes the above
problem by not setting 0 KEY_PTRS of journal key, if there is no bucket
available to reclaim.
Signed-off-by: Coly Li <colyli@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-25 00:48:33 +08:00
|
|
|
if (n) {
|
|
|
|
bkey_init(k);
|
|
|
|
SET_KEY_PTRS(k, n);
|
2013-03-24 07:11:31 +08:00
|
|
|
c->journal.blocks_free = c->sb.bucket_size >> c->block_bits;
|
bcache: never set KEY_PTRS of journal key to 0 in journal_reclaim()
In journal_reclaim() ja->cur_idx of each cache will be update to
reclaim available journal buckets. Variable 'int n' is used to count how
many cache is successfully reclaimed, then n is set to c->journal.key
by SET_KEY_PTRS(). Later in journal_write_unlocked(), a for_each_cache()
loop will write the jset data onto each cache.
The problem is, if all jouranl buckets on each cache is full, the
following code in journal_reclaim(),
529 for_each_cache(ca, c, iter) {
530 struct journal_device *ja = &ca->journal;
531 unsigned int next = (ja->cur_idx + 1) % ca->sb.njournal_buckets;
532
533 /* No space available on this device */
534 if (next == ja->discard_idx)
535 continue;
536
537 ja->cur_idx = next;
538 k->ptr[n++] = MAKE_PTR(0,
539 bucket_to_sector(c, ca->sb.d[ja->cur_idx]),
540 ca->sb.nr_this_dev);
541 }
542
543 bkey_init(k);
544 SET_KEY_PTRS(k, n);
If there is no available bucket to reclaim, the if() condition at line
534 will always true, and n remains 0. Then at line 544, SET_KEY_PTRS()
will set KEY_PTRS field of c->journal.key to 0.
Setting KEY_PTRS field of c->journal.key to 0 is wrong. Because in
journal_write_unlocked() the journal data is written in following loop,
649 for (i = 0; i < KEY_PTRS(k); i++) {
650-671 submit journal data to cache device
672 }
If KEY_PTRS field is set to 0 in jouranl_reclaim(), the journal data
won't be written to cache device here. If system crahed or rebooted
before bkeys of the lost journal entries written into btree nodes, data
corruption will be reported during bcache reload after rebooting the
system.
Indeed there is only one cache in a cache set, there is no need to set
KEY_PTRS field in journal_reclaim() at all. But in order to keep the
for_each_cache() logic consistent for now, this patch fixes the above
problem by not setting 0 KEY_PTRS of journal key, if there is no bucket
available to reclaim.
Signed-off-by: Coly Li <colyli@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-25 00:48:33 +08:00
|
|
|
}
|
2013-10-25 08:07:04 +08:00
|
|
|
out:
|
2013-03-24 07:11:31 +08:00
|
|
|
if (!journal_full(&c->journal))
|
|
|
|
__closure_wake_up(&c->journal.wait);
|
|
|
|
}
|
|
|
|
|
|
|
|
void bch_journal_next(struct journal *j)
|
|
|
|
{
|
|
|
|
atomic_t p = { 1 };
|
|
|
|
|
|
|
|
j->cur = (j->cur == j->w)
|
|
|
|
? &j->w[1]
|
|
|
|
: &j->w[0];
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The fifo_push() needs to happen at the same time as j->seq is
|
|
|
|
* incremented for last_seq() to be calculated correctly
|
|
|
|
*/
|
|
|
|
BUG_ON(!fifo_push(&j->pin, p));
|
|
|
|
atomic_set(&fifo_back(&j->pin), 1);
|
|
|
|
|
|
|
|
j->cur->data->seq = ++j->seq;
|
2014-02-20 11:48:26 +08:00
|
|
|
j->cur->dirty = false;
|
2013-03-24 07:11:31 +08:00
|
|
|
j->cur->need_write = false;
|
|
|
|
j->cur->data->keys = 0;
|
|
|
|
|
|
|
|
if (fifo_full(&j->pin))
|
|
|
|
pr_debug("journal_pin full (%zu)", fifo_used(&j->pin));
|
|
|
|
}
|
|
|
|
|
2015-07-20 21:29:37 +08:00
|
|
|
static void journal_write_endio(struct bio *bio)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
|
|
|
struct journal_write *w = bio->bi_private;
|
|
|
|
|
2017-06-03 15:38:06 +08:00
|
|
|
cache_set_err_on(bio->bi_status, w->c, "journal io error");
|
2013-10-09 06:50:46 +08:00
|
|
|
closure_put(&w->c->journal.io);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
2018-08-11 13:19:46 +08:00
|
|
|
static void journal_write(struct closure *cl);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
static void journal_write_done(struct closure *cl)
|
|
|
|
{
|
2013-10-09 06:50:46 +08:00
|
|
|
struct journal *j = container_of(cl, struct journal, io);
|
2013-03-24 07:11:31 +08:00
|
|
|
struct journal_write *w = (j->cur == j->w)
|
|
|
|
? &j->w[1]
|
|
|
|
: &j->w[0];
|
|
|
|
|
|
|
|
__closure_wake_up(&w->wait);
|
2018-09-27 23:41:46 +08:00
|
|
|
continue_at_nobarrier(cl, journal_write, bch_journal_wq);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
2013-12-17 07:27:25 +08:00
|
|
|
static void journal_write_unlock(struct closure *cl)
|
2018-03-19 08:36:32 +08:00
|
|
|
__releases(&c->journal.lock)
|
2013-12-17 07:27:25 +08:00
|
|
|
{
|
|
|
|
struct cache_set *c = container_of(cl, struct cache_set, journal.io);
|
|
|
|
|
|
|
|
c->journal.io_in_flight = 0;
|
|
|
|
spin_unlock(&c->journal.lock);
|
|
|
|
}
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
static void journal_write_unlocked(struct closure *cl)
|
2013-03-27 04:49:02 +08:00
|
|
|
__releases(c->journal.lock)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
2013-10-09 06:50:46 +08:00
|
|
|
struct cache_set *c = container_of(cl, struct cache_set, journal.io);
|
2013-03-24 07:11:31 +08:00
|
|
|
struct cache *ca;
|
|
|
|
struct journal_write *w = c->journal.cur;
|
|
|
|
struct bkey *k = &c->journal.key;
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int i, sectors = set_blocks(w->data, block_bytes(c)) *
|
2013-12-18 15:49:49 +08:00
|
|
|
c->sb.block_size;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
struct bio *bio;
|
|
|
|
struct bio_list list;
|
2018-08-11 13:19:45 +08:00
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
bio_list_init(&list);
|
|
|
|
|
|
|
|
if (!w->need_write) {
|
2013-12-17 07:27:25 +08:00
|
|
|
closure_return_with_destructor(cl, journal_write_unlock);
|
2015-03-06 23:37:46 +08:00
|
|
|
return;
|
2013-03-24 07:11:31 +08:00
|
|
|
} else if (journal_full(&c->journal)) {
|
|
|
|
journal_reclaim(c);
|
|
|
|
spin_unlock(&c->journal.lock);
|
|
|
|
|
|
|
|
btree_flush_write(c);
|
2018-09-27 23:41:46 +08:00
|
|
|
continue_at(cl, journal_write, bch_journal_wq);
|
2015-03-06 23:37:46 +08:00
|
|
|
return;
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
2013-12-18 15:49:49 +08:00
|
|
|
c->journal.blocks_free -= set_blocks(w->data, block_bytes(c));
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
w->data->btree_level = c->root->level;
|
|
|
|
|
|
|
|
bkey_copy(&w->data->btree_root, &c->root->key);
|
|
|
|
bkey_copy(&w->data->uuid_bucket, &c->uuid_bucket);
|
|
|
|
|
|
|
|
for_each_cache(ca, c, i)
|
|
|
|
w->data->prio_bucket[ca->sb.nr_this_dev] = ca->prio_buckets[0];
|
|
|
|
|
2013-11-01 06:46:42 +08:00
|
|
|
w->data->magic = jset_magic(&c->sb);
|
2013-03-24 07:11:31 +08:00
|
|
|
w->data->version = BCACHE_JSET_VERSION;
|
|
|
|
w->data->last_seq = last_seq(&c->journal);
|
|
|
|
w->data->csum = csum_set(w->data);
|
|
|
|
|
|
|
|
for (i = 0; i < KEY_PTRS(k); i++) {
|
|
|
|
ca = PTR_CACHE(c, k, i);
|
|
|
|
bio = &ca->journal.bio;
|
|
|
|
|
|
|
|
atomic_long_add(sectors, &ca->meta_sectors_written);
|
|
|
|
|
|
|
|
bio_reset(bio);
|
2013-10-12 06:44:27 +08:00
|
|
|
bio->bi_iter.bi_sector = PTR_OFFSET(k, i);
|
2017-08-24 01:10:32 +08:00
|
|
|
bio_set_dev(bio, ca->bdev);
|
2013-10-12 06:44:27 +08:00
|
|
|
bio->bi_iter.bi_size = sectors << 9;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
bio->bi_end_io = journal_write_endio;
|
|
|
|
bio->bi_private = w;
|
2016-06-06 03:32:05 +08:00
|
|
|
bio_set_op_attrs(bio, REQ_OP_WRITE,
|
2016-06-06 03:32:25 +08:00
|
|
|
REQ_SYNC|REQ_META|REQ_PREFLUSH|REQ_FUA);
|
2013-03-29 02:50:55 +08:00
|
|
|
bch_bio_map(bio, w->data);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2018-12-13 22:53:57 +08:00
|
|
|
trace_bcache_journal_write(bio, w->data->keys);
|
2013-03-24 07:11:31 +08:00
|
|
|
bio_list_add(&list, bio);
|
|
|
|
|
|
|
|
SET_PTR_OFFSET(k, i, PTR_OFFSET(k, i) + sectors);
|
|
|
|
|
|
|
|
ca->journal.seq[ca->journal.cur_idx] = w->data->seq;
|
|
|
|
}
|
|
|
|
|
bcache: never set KEY_PTRS of journal key to 0 in journal_reclaim()
In journal_reclaim() ja->cur_idx of each cache will be update to
reclaim available journal buckets. Variable 'int n' is used to count how
many cache is successfully reclaimed, then n is set to c->journal.key
by SET_KEY_PTRS(). Later in journal_write_unlocked(), a for_each_cache()
loop will write the jset data onto each cache.
The problem is, if all jouranl buckets on each cache is full, the
following code in journal_reclaim(),
529 for_each_cache(ca, c, iter) {
530 struct journal_device *ja = &ca->journal;
531 unsigned int next = (ja->cur_idx + 1) % ca->sb.njournal_buckets;
532
533 /* No space available on this device */
534 if (next == ja->discard_idx)
535 continue;
536
537 ja->cur_idx = next;
538 k->ptr[n++] = MAKE_PTR(0,
539 bucket_to_sector(c, ca->sb.d[ja->cur_idx]),
540 ca->sb.nr_this_dev);
541 }
542
543 bkey_init(k);
544 SET_KEY_PTRS(k, n);
If there is no available bucket to reclaim, the if() condition at line
534 will always true, and n remains 0. Then at line 544, SET_KEY_PTRS()
will set KEY_PTRS field of c->journal.key to 0.
Setting KEY_PTRS field of c->journal.key to 0 is wrong. Because in
journal_write_unlocked() the journal data is written in following loop,
649 for (i = 0; i < KEY_PTRS(k); i++) {
650-671 submit journal data to cache device
672 }
If KEY_PTRS field is set to 0 in jouranl_reclaim(), the journal data
won't be written to cache device here. If system crahed or rebooted
before bkeys of the lost journal entries written into btree nodes, data
corruption will be reported during bcache reload after rebooting the
system.
Indeed there is only one cache in a cache set, there is no need to set
KEY_PTRS field in journal_reclaim() at all. But in order to keep the
for_each_cache() logic consistent for now, this patch fixes the above
problem by not setting 0 KEY_PTRS of journal key, if there is no bucket
available to reclaim.
Signed-off-by: Coly Li <colyli@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cc: stable@vger.kernel.org
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-25 00:48:33 +08:00
|
|
|
/* If KEY_PTRS(k) == 0, this jset gets lost in air */
|
|
|
|
BUG_ON(i == 0);
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
atomic_dec_bug(&fifo_back(&c->journal.pin));
|
|
|
|
bch_journal_next(&c->journal);
|
|
|
|
journal_reclaim(c);
|
|
|
|
|
|
|
|
spin_unlock(&c->journal.lock);
|
|
|
|
|
|
|
|
while ((bio = bio_list_pop(&list)))
|
bcache: add CACHE_SET_IO_DISABLE to struct cache_set flags
When too many I/Os failed on cache device, bch_cache_set_error() is called
in the error handling code path to retire whole problematic cache set. If
new I/O requests continue to come and take refcount dc->count, the cache
set won't be retired immediately, this is a problem.
Further more, there are several kernel thread and self-armed kernel work
may still running after bch_cache_set_error() is called. It needs to wait
quite a while for them to stop, or they won't stop at all. They also
prevent the cache set from being retired.
The solution in this patch is, to add per cache set flag to disable I/O
request on this cache and all attached backing devices. Then new coming I/O
requests can be rejected in *_make_request() before taking refcount, kernel
threads and self-armed kernel worker can stop very fast when flags bit
CACHE_SET_IO_DISABLE is set.
Because bcache also do internal I/Os for writeback, garbage collection,
bucket allocation, journaling, this kind of I/O should be disabled after
bch_cache_set_error() is called. So closure_bio_submit() is modified to
check whether CACHE_SET_IO_DISABLE is set on cache_set->flags. If set,
closure_bio_submit() will set bio->bi_status to BLK_STS_IOERR and
return, generic_make_request() won't be called.
A sysfs interface is also added to set or clear CACHE_SET_IO_DISABLE bit
from cache_set->flags, to disable or enable cache set I/O for debugging. It
is helpful to trigger more corner case issues for failed cache device.
Changelog
v4, add wait_for_kthread_stop(), and call it before exits writeback and gc
kernel threads.
v3, change CACHE_SET_IO_DISABLE from 4 to 3, since it is bit index.
remove "bcache: " prefix when printing out kernel message.
v2, more changes by previous review,
- Use CACHE_SET_IO_DISABLE of cache_set->flags, suggested by Junhui.
- Check CACHE_SET_IO_DISABLE in bch_btree_gc() to stop a while-loop, this
is reported and inspired from origal patch of Pavel Vazharov.
v1, initial version.
Signed-off-by: Coly Li <colyli@suse.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Reviewed-by: Michael Lyle <mlyle@lyle.org>
Cc: Junhui Tang <tang.junhui@zte.com.cn>
Cc: Michael Lyle <mlyle@lyle.org>
Cc: Pavel Vazharov <freakpv@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-03-19 08:36:17 +08:00
|
|
|
closure_bio_submit(c, bio, cl);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
continue_at(cl, journal_write_done, NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void journal_write(struct closure *cl)
|
|
|
|
{
|
2013-10-09 06:50:46 +08:00
|
|
|
struct cache_set *c = container_of(cl, struct cache_set, journal.io);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
spin_lock(&c->journal.lock);
|
|
|
|
journal_write_unlocked(cl);
|
|
|
|
}
|
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
static void journal_try_write(struct cache_set *c)
|
2013-03-27 04:49:02 +08:00
|
|
|
__releases(c->journal.lock)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
2013-10-09 06:50:46 +08:00
|
|
|
struct closure *cl = &c->journal.io;
|
|
|
|
struct journal_write *w = c->journal.cur;
|
|
|
|
|
|
|
|
w->need_write = true;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-12-17 07:27:25 +08:00
|
|
|
if (!c->journal.io_in_flight) {
|
|
|
|
c->journal.io_in_flight = 1;
|
|
|
|
closure_call(cl, journal_write_unlocked, NULL, &c->cl);
|
|
|
|
} else {
|
2013-10-25 08:07:04 +08:00
|
|
|
spin_unlock(&c->journal.lock);
|
2013-12-17 07:27:25 +08:00
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
static struct journal_write *journal_wait_for_write(struct cache_set *c,
|
2018-08-11 13:19:44 +08:00
|
|
|
unsigned int nkeys)
|
2018-03-19 08:36:32 +08:00
|
|
|
__acquires(&c->journal.lock)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
2013-10-25 08:07:04 +08:00
|
|
|
size_t sectors;
|
|
|
|
struct closure cl;
|
2013-12-11 08:10:46 +08:00
|
|
|
bool wait = false;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
closure_init_stack(&cl);
|
|
|
|
|
|
|
|
spin_lock(&c->journal.lock);
|
|
|
|
|
|
|
|
while (1) {
|
|
|
|
struct journal_write *w = c->journal.cur;
|
|
|
|
|
|
|
|
sectors = __set_blocks(w->data, w->data->keys + nkeys,
|
2013-12-18 15:49:49 +08:00
|
|
|
block_bytes(c)) * c->sb.block_size;
|
2013-10-25 08:07:04 +08:00
|
|
|
|
|
|
|
if (sectors <= min_t(size_t,
|
|
|
|
c->journal.blocks_free * c->sb.block_size,
|
|
|
|
PAGE_SECTORS << JSET_BITS))
|
|
|
|
return w;
|
|
|
|
|
2013-12-11 08:10:46 +08:00
|
|
|
if (wait)
|
|
|
|
closure_wait(&c->journal.wait, &cl);
|
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
if (!journal_full(&c->journal)) {
|
2013-12-11 08:10:46 +08:00
|
|
|
if (wait)
|
|
|
|
trace_bcache_journal_entry_full(c);
|
2013-10-25 08:07:04 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* XXX: If we were inserting so many keys that they
|
|
|
|
* won't fit in an _empty_ journal write, we'll
|
|
|
|
* deadlock. For now, handle this in
|
|
|
|
* bch_keylist_realloc() - but something to think about.
|
|
|
|
*/
|
|
|
|
BUG_ON(!w->data->keys);
|
|
|
|
|
|
|
|
journal_try_write(c); /* unlocks */
|
|
|
|
} else {
|
2013-12-11 08:10:46 +08:00
|
|
|
if (wait)
|
|
|
|
trace_bcache_journal_full(c);
|
2013-10-25 08:07:04 +08:00
|
|
|
|
|
|
|
journal_reclaim(c);
|
|
|
|
spin_unlock(&c->journal.lock);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
btree_flush_write(c);
|
|
|
|
}
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
closure_sync(&cl);
|
|
|
|
spin_lock(&c->journal.lock);
|
2013-12-11 08:10:46 +08:00
|
|
|
wait = true;
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-10-09 06:50:46 +08:00
|
|
|
static void journal_write_work(struct work_struct *work)
|
|
|
|
{
|
|
|
|
struct cache_set *c = container_of(to_delayed_work(work),
|
|
|
|
struct cache_set,
|
|
|
|
journal.work);
|
|
|
|
spin_lock(&c->journal.lock);
|
2014-02-20 11:48:26 +08:00
|
|
|
if (c->journal.cur->dirty)
|
|
|
|
journal_try_write(c);
|
|
|
|
else
|
|
|
|
spin_unlock(&c->journal.lock);
|
2013-10-09 06:50:46 +08:00
|
|
|
}
|
|
|
|
|
2013-03-24 07:11:31 +08:00
|
|
|
/*
|
|
|
|
* Entry point to the journalling code - bio_insert() and btree_invalidate()
|
|
|
|
* pass bch_journal() a list of keys to be journalled, and then
|
|
|
|
* bch_journal() hands those same keys off to btree_insert_async()
|
|
|
|
*/
|
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
atomic_t *bch_journal(struct cache_set *c,
|
|
|
|
struct keylist *keys,
|
|
|
|
struct closure *parent)
|
2013-03-24 07:11:31 +08:00
|
|
|
{
|
|
|
|
struct journal_write *w;
|
2013-10-25 08:07:04 +08:00
|
|
|
atomic_t *ret;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2019-06-28 19:59:36 +08:00
|
|
|
/* No journaling if CACHE_SET_IO_DISABLE set already */
|
|
|
|
if (unlikely(test_bit(CACHE_SET_IO_DISABLE, &c->flags)))
|
|
|
|
return NULL;
|
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
if (!CACHE_SYNC(&c->sb))
|
|
|
|
return NULL;
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
w = journal_wait_for_write(c, bch_keylist_nkeys(keys));
|
2013-04-27 06:39:55 +08:00
|
|
|
|
2013-12-18 13:56:21 +08:00
|
|
|
memcpy(bset_bkey_last(w->data), keys->keys, bch_keylist_bytes(keys));
|
2013-10-25 08:07:04 +08:00
|
|
|
w->data->keys += bch_keylist_nkeys(keys);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
ret = &fifo_back(&c->journal.pin);
|
|
|
|
atomic_inc(ret);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
2013-10-25 08:07:04 +08:00
|
|
|
if (parent) {
|
|
|
|
closure_wait(&w->wait, parent);
|
2013-10-09 06:50:46 +08:00
|
|
|
journal_try_write(c);
|
2014-02-20 11:48:26 +08:00
|
|
|
} else if (!w->dirty) {
|
|
|
|
w->dirty = true;
|
2013-10-09 06:50:46 +08:00
|
|
|
schedule_delayed_work(&c->journal.work,
|
|
|
|
msecs_to_jiffies(c->journal_delay_ms));
|
|
|
|
spin_unlock(&c->journal.lock);
|
|
|
|
} else {
|
|
|
|
spin_unlock(&c->journal.lock);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
2013-10-25 08:07:04 +08:00
|
|
|
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
void bch_journal_meta(struct cache_set *c, struct closure *cl)
|
|
|
|
{
|
|
|
|
struct keylist keys;
|
|
|
|
atomic_t *ref;
|
|
|
|
|
|
|
|
bch_keylist_init(&keys);
|
|
|
|
|
|
|
|
ref = bch_journal(c, &keys, cl);
|
|
|
|
if (ref)
|
|
|
|
atomic_dec_bug(ref);
|
2013-03-24 07:11:31 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void bch_journal_free(struct cache_set *c)
|
|
|
|
{
|
|
|
|
free_pages((unsigned long) c->journal.w[1].data, JSET_BITS);
|
|
|
|
free_pages((unsigned long) c->journal.w[0].data, JSET_BITS);
|
|
|
|
free_fifo(&c->journal.pin);
|
|
|
|
}
|
|
|
|
|
|
|
|
int bch_journal_alloc(struct cache_set *c)
|
|
|
|
{
|
|
|
|
struct journal *j = &c->journal;
|
|
|
|
|
|
|
|
spin_lock_init(&j->lock);
|
bcache: performance improvement for btree_flush_write()
This patch improves performance for btree_flush_write() in following
ways,
- Use another spinlock journal.flush_write_lock to replace the very
hot journal.lock. We don't have to use journal.lock here, selecting
candidate btree nodes takes a lot of time, hold journal.lock here will
block other jouranling threads and drop the overall I/O performance.
- Only select flushing btree node from c->btree_cache list. When the
machine has a large system memory, mca cache may have a huge number of
cached btree nodes. Iterating all the cached nodes will take a lot
of CPU time, and most of the nodes on c->btree_cache_freeable and
c->btree_cache_freed lists are cleared and have need to flush. So only
travel mca list c->btree_cache to select flushing btree node should be
enough for most of the cases.
- Don't iterate whole c->btree_cache list, only reversely select first
BTREE_FLUSH_NR btree nodes to flush. Iterate all btree nodes from
c->btree_cache and select the oldest journal pin btree nodes consumes
huge number of CPU cycles if the list is huge (push and pop a node
into/out of a heap is expensive). The last several dirty btree nodes
on the tail of c->btree_cache list are earlest allocated and cached
btree nodes, they are relative to the oldest journal pin btree nodes.
Therefore only flushing BTREE_FLUSH_NR btree nodes from tail of
c->btree_cache probably includes the oldest journal pin btree nodes.
In my testing, the above change decreases 50%+ CPU consumption when
journal space is full. Some times IOPS drops to 0 for 5-8 seconds,
comparing blocking I/O for 120+ seconds in previous code, this is much
better. Maybe there is room to improve in future, but at this momment
the fix looks fine and performs well in my testing.
Signed-off-by: Coly Li <colyli@suse.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-28 19:59:59 +08:00
|
|
|
spin_lock_init(&j->flush_write_lock);
|
2013-10-09 06:50:46 +08:00
|
|
|
INIT_DELAYED_WORK(&j->work, journal_write_work);
|
2013-03-24 07:11:31 +08:00
|
|
|
|
|
|
|
c->journal_delay_ms = 100;
|
|
|
|
|
|
|
|
j->w[0].c = c;
|
|
|
|
j->w[1].c = c;
|
|
|
|
|
2019-06-28 19:59:54 +08:00
|
|
|
if (!(init_fifo(&j->pin, JOURNAL_PIN, GFP_KERNEL)) ||
|
2013-03-24 07:11:31 +08:00
|
|
|
!(j->w[0].data = (void *) __get_free_pages(GFP_KERNEL, JSET_BITS)) ||
|
|
|
|
!(j->w[1].data = (void *) __get_free_pages(GFP_KERNEL, JSET_BITS)))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|