Commit
075d5837bafc317e796548f721b8fbf7add6821d upstream.
The io_mapped_ubuf of a registered buffer has one bio_vec per page,
unless the pages coalesce into huge pages. Huge pages only count against
RLIMIT_MEMLOCK once per ring, so a buffer that maps the same 2MB folio
over and over in a way that defeats coalescing gets a full bvec array
per registration while only the one folio counts as locked. 1GB of that
registered in 100 slots took several times a 64MB memory.max, with 2.6MB
charged.
Allocate the io_mapped_ubuf and the io_rsrc_node with
GFP_KERNEL_ACCOUNT. An object reused from the ring caches stays charged
to whoever allocated it first, same as the io_kiocb cache.
Fixes: de2939388be5 ("io_uring: improve registered buffer accounting for huge pages")
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[axboe: 6.18 uses kvmalloc() + struct_size_t() rather than
kvmalloc_flex()]
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
int nr_bvecs)
{
if (nr_bvecs <= IO_CACHED_BVECS_SEGS)
- return io_cache_alloc(&ctx->imu_cache, GFP_KERNEL);
+ return io_cache_alloc(&ctx->imu_cache, GFP_KERNEL_ACCOUNT);
return kvmalloc(struct_size_t(struct io_mapped_ubuf, bvec, nr_bvecs),
- GFP_KERNEL);
+ GFP_KERNEL_ACCOUNT);
}
static void io_free_imu(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu)
{
struct io_rsrc_node *node;
- node = io_cache_alloc(&ctx->node_cache, GFP_KERNEL);
+ node = io_cache_alloc(&ctx->node_cache, GFP_KERNEL_ACCOUNT);
if (node) {
node->type = type;
node->refs = 1;