Discussion:
async model for fuse lowlevel api
(too old to reply)
Marlon de Boer
2010-11-26 15:16:20 UTC
Permalink
Hi,

I've been searching around on the list but couldn't find a straight
answer to the following, I want to know if it's possible to have a async
io model for fuse, I want to do io based on libevent.

My guess was I could start a session and attach channels to it.
According to fuse_lowlevel.h I can only add 1 channel to a session,
tested it and indeed didn't work.

So my second thought was to create multiple sessions and add a single
channel to it. Then get the channel file descriptor and let libevent
monitor it. This approach didn't work either. I tried to do this:

if ((se = fuse_lowlevel_new(NULL, &liops_oper, sizeof(liops_oper),
NULL)) != NULL) {
if (fuse_set_signal_handlers(se) != -1) {
fuse_session_add_chan(se, ch);
}
}

if ((se2 = fuse_lowlevel_new(NULL, &liops_oper, sizeof(liops_oper),
NULL)) != NULL) {
if (fuse_set_signal_handlers(se2) != -1) {
fuse_session_add_chan(se2, ch2);
}
}

But my program just hangs then.

I wonder if anybody can tell me if it's possible to use a polling
mechanism like libevent with fuse and give me some directions how to set
this up, I'm especially interested how to setup multiple channels so I
can put the fd in libevent to monitor.

I looked at
http://permalink.gmane.org/gmane.comp.file-systems.fuse.devel/8922 and
http://old.nabble.com/Re:-Asynchronous-I-O-in-FUSE-td22934501.html but I
can't figure out how to start multiple channels which can be put into
libevent.

If somebody could explain how the sessions and channels work and how
they interact with (/dev/)fuse that would be great as well!

Regards
Marlon
Stef Bon
2010-11-26 20:34:03 UTC
Permalink
I guess you do not have more channels than one to do aio Why??

Off topic, the rest of the world apart from Holland can ognore this:

Looking at your name your dutch and email. I'm too, living in
Voorburg. Am I right?
Post by Marlon de Boer
Hi,
I've been searching around on the list but couldn't find a straight
answer to the following, I want to know if it's possible to have a async
io model for fuse, I want to do io based on libevent.
My guess was I could start a session and attach channels to it.
According to fuse_lowlevel.h I can only add 1 channel to a session,
tested it and indeed didn't work.
So my second thought was to create multiple sessions and add a single
channel to it. Then get the channel file descriptor and let libevent
 if ((se = fuse_lowlevel_new(NULL, &liops_oper, sizeof(liops_oper),
NULL)) != NULL) {
       if (fuse_set_signal_handlers(se) != -1) {
           fuse_session_add_chan(se, ch);
       }
   }
   if ((se2 = fuse_lowlevel_new(NULL, &liops_oper, sizeof(liops_oper),
NULL)) != NULL) {
       if (fuse_set_signal_handlers(se2) != -1) {
           fuse_session_add_chan(se2, ch2);
       }
   }
But my program just hangs then.
I wonder if anybody can tell me if it's possible to use a polling
mechanism like libevent with fuse and give me some directions how to set
this up, I'm especially interested how to setup multiple channels so I
can put the fd in libevent to monitor.
I looked at
http://permalink.gmane.org/gmane.comp.file-systems.fuse.devel/8922 and
http://old.nabble.com/Re:-Asynchronous-I-O-in-FUSE-td22934501.html but I
can't figure out how to start multiple channels which can be put into
libevent.
If somebody could explain how the sessions and channels work and how
they interact with (/dev/)fuse that would be great as well!
Regards
Marlon
------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
fuse-devel mailing list
https://lists.sourceforge.net/lists/listinfo/fuse-devel
Marlon de Boer
2010-11-27 09:59:27 UTC
Permalink
Post by Stef Bon
I guess you do not have more channels than one to do aio Why??
I want to spawn a fixed amount of read workers which don't have to be
separate threads, the standard pthread model just spawns an extra thread
for reads / writes and keeps 10 spare threads.

In my model those read workers still need an separate channel to
communicate I guess? What I want is to have an poll instance listing for
new read requests, that read request should be forwarded to an pool of
read workers doing the actual work.
Post by Stef Bon
Looking at your name your dutch and email. I'm too, living in
Voorburg. Am I right?
Yes I'm living in the Netherlands too, working for the social network Hyves.
Stef Bon
2010-11-27 12:15:44 UTC
Permalink
Post by Marlon de Boer
Post by Stef Bon
I guess you do not have more channels than one to do aio Why??
I want to spawn a fixed amount of read workers which don't have to be
separate threads, the standard pthread model just spawns an extra thread
for reads / writes and keeps 10 spare threads.
In my model those read workers still need an separate channel to
communicate I guess? What I want is to have an poll instance listing for
new read requests, that read request should be forwarded to an pool of
read workers doing the actual work.
Well, first of all, FUSE supports threads. So I guess you do not have
to hack FUSE to get multiple threads.

And, in recent posts you'll see aio discussed. That's also possible
when not using threads.
You'll have to change the mainloop, and rewirite it using select/poll/epoll.
The FUSE channnel is also represented by an fd, and is one of the fd's
epoll (or poll or select, it's your choice). Another fd is the fd
where aio operations are writing to when ready.

I'm very interested in making my own fs support aio, but I'm still
reading and testing a but here. I'm not that brilliant programmer, so
I have te read, write test programs etc before I can figure out how
something works.

But I know the big picture. The normal (blocking) read command
(myfs_read) for example (or write) will (in the lowlevel interface,
that's important, you are writing your fs using that interface do
you?) do the read request, using a buffer, "feed back" this result to
fuse using the fuse_reply_buff (or related) calls, and
will return. This return will mean that the fs is returning to the
mainloop, ready for the next request.

Now this is going to make the fs busy doing the read, and in the
meantime it's not able to handle other requests. So this is blocking.
These request are not lost, they are just waiting in the queue.

The whole idea with aio is to make the read command above use aio, and
that means it first
initialises an aio read command, and then launches it, together with
the fuse request handle and the event fd. Now this starts the aio
somehwere in the background by the kernel, and the read command
myfs_read can just return to the mainloop, while the real read is done
in the background.
This will make your fs open to new requests, while the command isn't
completed yet.
The aio read command in the background will do the actual work, and
will report back to the event fd, which you'll have to add to the fd's
your mainloop will listen to.

It's the task of the "aio command in background" (or io callback) to
report back the result to the eventfd, and write data in way the
listner on that fs (the mainloop) will recknoginze it and is able to
do something with it.

When there is data on that fd, which means the command has completed,
or there has been an error, the mainloop will read this, and launch
the right command (fuse_reply_...)

Read manpage io and eventfd. I haven't been able to write a good
construction though. Keep the maillist informed.

I hope this helps you,

Stef Bon
Marlon de Boer
2010-11-29 16:25:03 UTC
Permalink
Post by Stef Bon
Post by Marlon de Boer
Post by Stef Bon
I guess you do not have more channels than one to do aio Why??
I want to spawn a fixed amount of read workers which don't have to be
separate threads, the standard pthread model just spawns an extra thread
for reads / writes and keeps 10 spare threads.
In my model those read workers still need an separate channel to
communicate I guess? What I want is to have an poll instance listing for
new read requests, that read request should be forwarded to an pool of
read workers doing the actual work.
Well, first of all, FUSE supports threads. So I guess you do not have
to hack FUSE to get multiple threads.
I don't like the fact that fuse is spawning and killing threads whenever
a burst of reads occurs. I want a dedicated pool, and I'm not sure if I
even need threads with an async io model (cpu wise).
Post by Stef Bon
And, in recent posts you'll see aio discussed. That's also possible
when not using threads.
You'll have to change the mainloop, and rewirite it using select/poll/epoll.
The FUSE channnel is also represented by an fd, and is one of the fd's
epoll (or poll or select, it's your choice). Another fd is the fd
where aio operations are writing to when ready.
This part I understand, I've written a few programs that do async-io
based on epoll / libevent. I just don't see how I can create multiple
channels (fd's) that I can put into libevent.

I can create channels with fuse_chan_new, but I can only add 1 channel
to a session. Having multiple sessions to a mountpoint doesn't work.

<snip>
Post by Stef Bon
Read manpage io and eventfd. I haven't been able to write a good
construction though. Keep the maillist informed.
I hope this helps you,
Stef Bon
The whole io and event stuff is clear, I just don't know how to spawn
multiple channels in fuse, get the fd and put it into a poll mechanism.

Regards
Marlon
Goswin von Brederlow
2010-11-30 14:06:42 UTC
Permalink
Post by Marlon de Boer
Post by Stef Bon
Post by Marlon de Boer
Post by Stef Bon
I guess you do not have more channels than one to do aio Why??
I want to spawn a fixed amount of read workers which don't have to be
separate threads, the standard pthread model just spawns an extra thread
for reads / writes and keeps 10 spare threads.
In my model those read workers still need an separate channel to
communicate I guess? What I want is to have an poll instance listing for
new read requests, that read request should be forwarded to an pool of
read workers doing the actual work.
Well, first of all, FUSE supports threads. So I guess you do not have
to hack FUSE to get multiple threads.
I don't like the fact that fuse is spawning and killing threads whenever
a burst of reads occurs. I want a dedicated pool, and I'm not sure if I
even need threads with an async io model (cpu wise).
Post by Stef Bon
And, in recent posts you'll see aio discussed. That's also possible
when not using threads.
You'll have to change the mainloop, and rewirite it using select/poll/epoll.
The FUSE channnel is also represented by an fd, and is one of the fd's
epoll (or poll or select, it's your choice). Another fd is the fd
where aio operations are writing to when ready.
This part I understand, I've written a few programs that do async-io
based on epoll / libevent. I just don't see how I can create multiple
channels (fd's) that I can put into libevent.
I can create channels with fuse_chan_new, but I can only add 1 channel
to a session. Having multiple sessions to a mountpoint doesn't work.
You don't. You create one. The low-level interface is basically
asynchron. You get requests and you send replies in any order you
like. A channel isn't blocked with a request until it is replied to so
if the current request needs to wait you just poll for the fuse FD
and the IO any pending request is waiting for.

I suggest trying libaio and doing away with threads alltogether. Or
keep the threads and modify the existing multithreaded loop to use your
thread pools. [doesn't the current MT loop keep a thread pool already?]

But don't mix a single threaded loop with a thread pool. I think that
just added unneccessary complexity and communication overhead.
Post by Marlon de Boer
<snip>
Post by Stef Bon
Read manpage io and eventfd. I haven't been able to write a good
construction though. Keep the maillist informed.
I hope this helps you,
Stef Bon
The whole io and event stuff is clear, I just don't know how to spawn
multiple channels in fuse, get the fd and put it into a poll mechanism.
Regards
Marlon
MfG
Goswin
Stef Bon
2010-12-01 10:07:03 UTC
Permalink
Post by Goswin von Brederlow
Post by Marlon de Boer
I can create channels with fuse_chan_new, but I can only add 1 channel
to a session. Having multiple sessions to a mountpoint doesn't work.
You don't. You create one. The low-level interface is basically
asynchron. You get requests and you send replies in any order you
like. A channel isn't blocked with a request until it is replied to so
if the current request needs to wait you just poll for the fuse FD
and the IO any pending request is waiting for.
I suggest trying libaio and doing away with threads alltogether.  Or
keep the threads and modify the existing multithreaded loop to use your
thread pools. [doesn't the current MT loop keep a thread pool already?]
Maybe use libevent for that also.

Look for earlier posts. I've seen combinations of libevent and fuse before.

Stef
John Kozak
2010-12-01 12:09:35 UTC
Permalink
I've used the fuse lowlevel api with gevent (python libevent greenlet
wrapper). Works nicely; as mentioned upthread the lowlevel api is
already async, so no multiplicity of channels or sessions is needed. I
did need to hack libevent into the event loop, of course.
--
John
Continue reading on narkive:
Loading...