NFS and inotify

[Phil Endecott]

Dear NFS experts,

I am trying to understand how Linux NFS does/could interact with
inotify. Please let me know how close this is to the truth:

- NFSv3 or older can't easily provide any sort of notification because
of statelessness.

- NFSv4 has state, so the protocol could perhaps support some kind of notification.

- The current Linux NFSv4 implementation doesn't support any notification.

- The inotify kernel API lacks some features needed to implement NFS
notification. (I'm basing this on Asser Fem??'s observation that he
couldn't integrate CIFS with inotify, e.g. http://lists.samba.org/archive/linux-cifs-client/2005-August/000937.html)

- Asking inotify to monitor an NFS file results in an "OK" response,
and no events, rather than an error. (Returing an error would make it
easier for an application to switch to a fallback, e.g. polling; what's
the best way to determine whether a particular file is on NFS or not?)


Is that accurate?


Many thanks,

Phil.

[Trond Myklebust]

Post by Phil Endecott
Dear NFS experts,
I am trying to understand how Linux NFS does/could interact with
- NFSv3 or older can't easily provide any sort of notification because
of statelessness.
- NFSv4 has state, so the protocol could perhaps support some kind of notification.
- The current Linux NFSv4 implementation doesn't support any notification.
- The inotify kernel API lacks some features needed to implement NFS
notification. (I'm basing this on Asser Fem?'s observation that he
couldn't integrate CIFS with inotify, e.g. http://lists.samba.org/archive/linux-cifs-client/2005-August/000937.html)
- Asking inotify to monitor an NFS file results in an "OK" response,
and no events, rather than an error. (Returing an error would make it
easier for an application to switch to a fallback, e.g. polling; what's
the best way to determine whether a particular file is on NFS or not?)
Is that accurate?

You forgot the main thing: neither the NFSv2, v3, or v4 protocol have
any mechanism whatsoever for supporting inotify (which is the main
reason why none of the implementations support it).

Cheers
Trond

[J. Bruce Fields]

Post by Trond Myklebust

Post by Phil Endecott
Dear NFS experts,
I am trying to understand how Linux NFS does/could interact with
- NFSv3 or older can't easily provide any sort of notification because
of statelessness.
- NFSv4 has state, so the protocol could perhaps support some kind of notification.
- The current Linux NFSv4 implementation doesn't support any notification.
- The inotify kernel API lacks some features needed to implement NFS
notification. (I'm basing this on Asser Fem?'s observation that he
couldn't integrate CIFS with inotify, e.g. http://lists.samba.org/archive/linux-cifs-client/2005-August/000937.html)
- Asking inotify to monitor an NFS file results in an "OK" response,
and no events, rather than an error. (Returing an error would make it
easier for an application to switch to a fallback, e.g. polling; what's
the best way to determine whether a particular file is on NFS or not?)
Is that accurate?

You forgot the main thing: neither the NFSv2, v3, or v4 protocol have
any mechanism whatsoever for supporting inotify (which is the main
reason why none of the implementations support it).

But 4.1 has something:

http://nfsv4-editor.org/drafts/drafts.html

(Look for "directory delegations".) I suspect it would still be
difficult to implement inotify over that, but I don't know if anyone's
really studied the question.

--b.

[Phil Endecott]

[snip]

Post by Trond Myklebust

Post by Phil Endecott
- NFSv4 has state, so the protocol could perhaps support some kind of notification.

[snip]

Post by Trond Myklebust
neither the NFSv2, v3, or v4 protocol have
any mechanism whatsoever for supporting inotify (which is the main
reason why none of the implementations support it).

My understanding was that NFSv4 clients receive some sort of
notification ("callback"?) from the server that they use to invalidate
their caches. Is this not true? Can this not be used for [some subset
of] inotify?

Phil.

[J. Bruce Fields]

Post by Phil Endecott
[snip]

Post by Trond Myklebust

Post by Phil Endecott
- NFSv4 has state, so the protocol could perhaps support some kind of notification.

[snip]

Post by Trond Myklebust
neither the NFSv2, v3, or v4 protocol have
any mechanism whatsoever for supporting inotify (which is the main
reason why none of the implementations support it).

My understanding was that NFSv4 clients receive some sort of
notification ("callback"?) from the server that they use to invalidate
their caches. Is this not true? Can this not be used for [some subset
of] inotify?

No, it wouldn't work. Those callbacks only go to clients that hold
delegations on a file. Delegations are given out at the server's whim
(and in NFSv4.0 only in response to opens), and there's just one
callback to break the delegation after which the client gets no more
callbacks until it gets another delegation on the file (which the server
is unlikely to grant if there was a conflicting access so recently).

The directory delegations in NFSv4.1 might give you some of what you
want, but I don't know.

--b.

[Phil Endecott]

Post by J. Bruce Fields

Post by Phil Endecott
[snip]

Post by Trond Myklebust

Post by Phil Endecott
- NFSv4 has state, so the protocol could perhaps support some kind of notification.

[snip]

Post by Trond Myklebust
neither the NFSv2, v3, or v4 protocol have
any mechanism whatsoever for supporting inotify (which is the main
reason why none of the implementations support it).

My understanding was that NFSv4 clients receive some sort of
notification ("callback"?) from the server that they use to invalidate
their caches. Is this not true? Can this not be used for [some subset
of] inotify?

No, it wouldn't work. Those callbacks only go to clients that hold
delegations on a file. Delegations are given out at the server's whim
(and in NFSv4.0 only in response to opens), and there's just one
callback to break the delegation after which the client gets no more
callbacks until it gets another delegation on the file (which the server
is unlikely to grant if there was a conflicting access so recently).

Thanks, that's clear now.

Post by J. Bruce Fields
The directory delegations in NFSv4.1 might give you some of what you
want, but I don't know.

Right. I'm a bit surprised that this hasn't got much attention
before, since GUI directory browsers are so common.

Any thoughts about my observation that Linux happily accepts an inotify
watch request for an NFS file, even though it can't honour it? I
should perhaps ask about this on LKML.


Cheers,

Phil.

[J. Bruce Fields]

Post by Phil Endecott

Post by J. Bruce Fields

Post by Phil Endecott
[snip]

Post by Trond Myklebust

Post by Phil Endecott
- NFSv4 has state, so the protocol could perhaps support some kind of notification.

[snip]

Post by Trond Myklebust
neither the NFSv2, v3, or v4 protocol have
any mechanism whatsoever for supporting inotify (which is the main
reason why none of the implementations support it).

My understanding was that NFSv4 clients receive some sort of notification
("callback"?) from the server that they use to invalidate their caches.
Is this not true? Can this not be used for [some subset of] inotify?

No, it wouldn't work. Those callbacks only go to clients that hold
delegations on a file. Delegations are given out at the server's whim
(and in NFSv4.0 only in response to opens), and there's just one
callback to break the delegation after which the client gets no more
callbacks until it gets another delegation on the file (which the server
is unlikely to grant if there was a conflicting access so recently).

Thanks, that's clear now.

Post by J. Bruce Fields
The directory delegations in NFSv4.1 might give you some of what you
want, but I don't know.

Right. I'm a bit surprised that this hasn't got much attention before,
since GUI directory browsers are so common.

Well, if you know of someone that's intrested, one good place to start
might be with a careful study of the 4.1 draft to figure out whether it
would provide adequate protocol support.

Post by Phil Endecott
Any thoughts about my observation that Linux happily accepts an inotify
watch request for an NFS file, even though it can't honour it?

That does sound like unhelpful behavior.

--b.

[Trond Myklebust]

Post by Phil Endecott
[snip]

Post by Trond Myklebust

Post by Phil Endecott
- NFSv4 has state, so the protocol could perhaps support some kind of notification.

[snip]

Post by Trond Myklebust
neither the NFSv2, v3, or v4 protocol have
any mechanism whatsoever for supporting inotify (which is the main
reason why none of the implementations support it).

My understanding was that NFSv4 clients receive some sort of
notification ("callback"?) from the server that they use to invalidate
their caches. Is this not true? Can this not be used for [some subset
of] inotify?

No. You are thinking of delegations (the equivalent of CIFS op-locks).
The main point about NFSv4 delegations is that the server is never
required by the protocol to give them out. In fact, in many cases, it
simply cannot give them out, either because some other client holds a
conflicting share open on the file, or because the client requesting the
delegation doesn't have a callback channel that the server can use. In
addition, delegations may only be given out for regular files. They
cannot therefore be used to implement dnotify or inotify.

OTOH, as Bruce mentioned, NFSv4 minor version 1 does add something
called directory notifications, which should be sufficient to implement
most of the features provided by dnotify(). Whether or not they will
allow for a full implementation of inotify is less certain.

Trond