Boris Protopopov
2012-01-18 15:24:05 UTC
Hi, everybody,
I have a filesystem based on FUSE low-level APIs. I do re-use inode numbers when they become available, and my inode allocation algorithm tends to reuse inode number that have just been freed. It works fine generally, however, when I run tests that issue lots of metadata operations (create, link, unlink, mkdir, rmdir, symlink), I get sporadic EBUSY errors from mkdir().
The problems only occur when I run multiple processes (tests performing same steps concurrently) agains a single filesystem; single process always works fine. My guess multiple processes cause sufficiently fast inode number reuse (one process frees inode number, and another allocates it quickly). I've been trying to figure out what the issue might be, looked through the kernel module source code (this is where EBUSY comes from), and found that EBUSY seems to be returned when FUSE sees more than one link to a directory inode. This seems reasonable, but I cannot see how this could happen with the tests I am running.
I have couple questions that I hope someone could answer:
1) what is the proper use of the fuse_lowlevel_notify_inval_inode()/fuse_lowlevel_notify_inval_entry()?
It is my understanding that unless the inodes/directories are modified in some way other than through local VFS interfaces/fuse.ko, I do not need to use these. I believe fuse.ko code does all the invalidation that is needed for all the operations that come through VFS, unless I missed something. Is this correct ?
2) What is the meaning of 'nodeid' as opposed to inode number ?
It appears that nodeids are the numbers returned to fuse.ko in the struct stat by the fuse_reply_enty() and similar calls. My assumption is that the inode numbers (in struct inode, part of fuse_inode) are supposed to be equal to nodeids, nowever, I am confused by the following comment in fs/fuse/dir.c::fuse_link():
...
/* Contrary to "normal" filesystems it can happen that link
makes two "logical" inodes point to the same "physical"
inode. We invalidate the attributes of the old one, so it
will reflect changes in the backing inode (link count,
etc.)
if (!err || err == -EINTR)
fuse_invalidate_attr(inode);
return err;
...
Does anyone understand what this means ?
What are the "logical" vs "physical" inodes and do they have to do with nodeids vs inode numbers ?
Going a step further, if this situation with two logical inodes occurs, could this lead to the EBUSY problem discussed above ?
Best regards, Boris Protopopov.
I have a filesystem based on FUSE low-level APIs. I do re-use inode numbers when they become available, and my inode allocation algorithm tends to reuse inode number that have just been freed. It works fine generally, however, when I run tests that issue lots of metadata operations (create, link, unlink, mkdir, rmdir, symlink), I get sporadic EBUSY errors from mkdir().
The problems only occur when I run multiple processes (tests performing same steps concurrently) agains a single filesystem; single process always works fine. My guess multiple processes cause sufficiently fast inode number reuse (one process frees inode number, and another allocates it quickly). I've been trying to figure out what the issue might be, looked through the kernel module source code (this is where EBUSY comes from), and found that EBUSY seems to be returned when FUSE sees more than one link to a directory inode. This seems reasonable, but I cannot see how this could happen with the tests I am running.
I have couple questions that I hope someone could answer:
1) what is the proper use of the fuse_lowlevel_notify_inval_inode()/fuse_lowlevel_notify_inval_entry()?
It is my understanding that unless the inodes/directories are modified in some way other than through local VFS interfaces/fuse.ko, I do not need to use these. I believe fuse.ko code does all the invalidation that is needed for all the operations that come through VFS, unless I missed something. Is this correct ?
2) What is the meaning of 'nodeid' as opposed to inode number ?
It appears that nodeids are the numbers returned to fuse.ko in the struct stat by the fuse_reply_enty() and similar calls. My assumption is that the inode numbers (in struct inode, part of fuse_inode) are supposed to be equal to nodeids, nowever, I am confused by the following comment in fs/fuse/dir.c::fuse_link():
...
/* Contrary to "normal" filesystems it can happen that link
makes two "logical" inodes point to the same "physical"
inode. We invalidate the attributes of the old one, so it
will reflect changes in the backing inode (link count,
etc.)
if (!err || err == -EINTR)
fuse_invalidate_attr(inode);
return err;
...
Does anyone understand what this means ?
What are the "logical" vs "physical" inodes and do they have to do with nodeids vs inode numbers ?
Going a step further, if this situation with two logical inodes occurs, could this lead to the EBUSY problem discussed above ?
Best regards, Boris Protopopov.