[fuse-devel] Read performance issue when upgrading from FUSE 2.9.4 to 3.0.2 (readahead not working?)

Discussion:

David Sorber

2017-06-07 15:09:05 UTC

I've been working on a FUSE-based file system using FUSE 2.9.4 on Ubuntu
14.04 for a while now. It works well and is reasonably stable. Recently I
decided it might be a good idea to upgrade to FUSE 3. I made the necessary
interface changes, installed FUSE 3, and everything works nicely.

However, I noticed a performance regression after upgrading to FUSE 3. I
wrote a very simple test utility that calls fgets() in a loop and iterates
over a text file that is ~120 MB. Using the previous revision that uses
FUSE 2, the test utility runs in ~200 ms. When I switch to the FUSE 3
revision the same utility now takes ~30,000 ms. Note that the only
modifications I made were interface modifications to use FUSE 3 instead of
FUSE 2, no other code was changed. I have both FUSE 2.9.4 and FUSE 3.0.2
installed side-by-side on the same machine for testing.

I added some additional debug logging and upon closer inspection I noticed
that for the FUSE 2 revision I see ~4-5 read requests per millisecond. For
the FUSE 3 revision I see only 1 read request every ~20 milliseconds. The
time to complete each read appears to be the same across both revisions.
This leads me to wonder (and maybe I'm totally wrong about this...) if
somehow the kernel block layer's readahead isn't working correctly, or
isn't configured when I'm using FUSE 3? Does anyone have any ideas as to
what may be wrong? or other things to look at?

Below is the initial startup debug output for both revisions:

FUSE library version: 2.9.4
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
FUSE library version: 2.9.4
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.23
flags=0x0003f7fb
max_readahead=0x00020000
INIT: 7.19
flags=0x00000031
max_readahead=0x00020000
max_write=0x00020000
max_background=0
congestion_threshold=0
unique: 1, success, outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 7065
getattr /
unique: 2, success, outsize: 120

FUSE library version: 3.0.2
nullpath_ok: 0
FUSE library version: 3.0.2
nullpath_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.23
flags=0x0003f7fb
max_readahead=0x00020000
INIT: 7.26
flags=0x0000d031
max_readahead=0x00020000
max_write=0x00020000
max_background=0
congestion_threshold=0
time_gran=0
unique: 1, success, outsize: 80
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 8531
getattr[NULL] /
unique: 2, success, outsize: 120

Things look to be configured identically as far as I can tell.

Any thoughts or suggestions would be appreciated.

Thanks,
-David

Nikolaus Rath

2017-06-07 17:02:31 UTC

Permalink

Post by David Sorber
I've been working on a FUSE-based file system using FUSE 2.9.4 on Ubuntu
14.04 for a while now. It works well and is reasonably stable. Recently I
decided it might be a good idea to upgrade to FUSE 3. I made the necessary
interface changes, installed FUSE 3, and everything works nicely.
However, I noticed a performance regression after upgrading to FUSE 3. I
wrote a very simple test utility that calls fgets() in a loop and iterates
over a text file that is ~120 MB. Using the previous revision that uses
FUSE 2, the test utility runs in ~200 ms. When I switch to the FUSE 3
revision the same utility now takes ~30,000 ms. Note that the only
modifications I made were interface modifications to use FUSE 3 instead of
FUSE 2, no other code was changed. I have both FUSE 2.9.4 and FUSE 3.0.2
installed side-by-side on the same machine for testing.

Are you setting up the capabilities in the same way? Note that the
defaults have changed. In particular, I'd look at (from
fuse_common.h):

/**
* Traditionally, while a file is open the FUSE kernel module only
* asks the filesystem for an update of the file's attributes when a
* client attempts to read beyond EOF. This is unsuitable for
* e.g. network filesystems, where the file contents may change
* without the kernel knowing about it.
*
* If this flag is set, FUSE will check the validity of the attributes
* on every read. If the attributes are no longer valid (i.e., if the
* *attr_timeout* passed to fuse_reply_attr() or set in `struct
* fuse_entry_param` has passed), it will first issue a `getattr`
* request. If the new mtime differs from the previous value, any
* cached file *contents* will be invalidated as well.
*
* This flag should always be set when available. If all file changes
* go through the kernel, *attr_timeout* should be set to zero to
* avoid unneccessary getattr() calls.
*
* This feature is enabled by default when supported by the kernel.
*/
#define FUSE_CAP_AUTO_INVAL_DATA (1 << 12)

Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

David Sorber

2017-06-07 17:38:50 UTC

Permalink

Nikolaus,

Thanks for pointing me in the right direction. FUSE_CAP_AUTO_INVAL_DATA
appears to be the culprit. (Although my FUSE 3 revision is now running in
~300 ms instead of the ~200 ms for the FUSE 2 revision... I guess I can
live with that for now).

I am setting attr_timeout to zero (all my file changes do indeed go through
the kernel), but I guess the attribute validity check on every read was
causing the slowdown. I missed that detail.

Thanks again.

-David