AcoSail/kernel_optimize_test
8
0
Fork 0
forked from luck/tmp_suning_uos_patched
Code Pull Requests Activity

bpf, lpm: make longest_prefix_match() faster

Browse Source 
At LPC 2018 in Vancouver, Vlad Dumitrescu mentioned that longest_prefix_match()
has a high cost [1].

One reason for that cost is a loop handling one byte at a time.

We can handle more bytes at a time, if enough attention is paid
to endianness.

I was able to remove ~55 % of longest_prefix_match() cpu costs.

[1] https://linuxplumbersconf.org/event/2/contributions/88/attachments/76/87/lpc-bpf-2018-shaping.pdf

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Vlad Dumitrescu <vladum@google.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
This commit is contained in:
Eric Dumazet 2018-11-21 21:39:52 -08:00 committed by Daniel Borkmann
parent e4b0c94bd2
commit 8d75839b84
1 changed files with 49 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

59
kernel/bpf/lpm_trie.c
View File

@ -168,20 +168,59 @@ static size_t longest_prefix_match(const struct lpm_trie *trie,
const struct lpm_trie_node *node,
const struct bpf_lpm_trie_key *key)
{
size_t prefixlen = 0;
size_t i;
u32 limit = min(node->prefixlen, key->prefixlen);
u32 prefixlen = 0, i = 0;
for (i = 0; i < trie->data_size; i++) {
size_t b;
BUILD_BUG_ON(offsetof(struct lpm_trie_node, data) % sizeof(u32));
BUILD_BUG_ON(offsetof(struct bpf_lpm_trie_key, data) % sizeof(u32));
b = 8 - fls(node->data[i] ^ key->data[i]);
prefixlen += b;
#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && defined(CONFIG_64BIT)
if (prefixlen >= node->prefixlen || prefixlen >= key->prefixlen)
return min(node->prefixlen, key->prefixlen);
/* data_size >= 16 has very small probability.
* We do not use a loop for optimal code generation.
*/
if (trie->data_size >= 8) {
u64 diff = be64_to_cpu(*(__be64 *)node->data ^
*(__be64 *)key->data);
if (b < 8)
break;
prefixlen = 64 - fls64(diff);
if (prefixlen >= limit)
return limit;
if (diff)
return prefixlen;
i = 8;
}
#endif
while (trie->data_size >= i + 4) {
u32 diff = be32_to_cpu(*(__be32 *)&node->data[i] ^
*(__be32 *)&key->data[i]);
prefixlen += 32 - fls(diff);
if (prefixlen >= limit)
return limit;
if (diff)
return prefixlen;
i += 4;
}
if (trie->data_size >= i + 2) {
u16 diff = be16_to_cpu(*(__be16 *)&node->data[i] ^
*(__be16 *)&key->data[i]);
prefixlen += 16 - fls(diff);
if (prefixlen >= limit)
return limit;
if (diff)
return prefixlen;
i += 2;
}
if (trie->data_size >= i + 1) {
prefixlen += 8 - fls(node->data[i] ^ key->data[i]);
if (prefixlen >= limit)
return limit;
}
return prefixlen;