Discussion:
READ/RENEW loop bug
Steven Brudenell
2010-01-17 05:39:46 UTC
Permalink
Hello list,

I'm having a pretty nasty issue with nfs4. I work in a Linux
environment where a lot of software is run off of a central nfs4
server (i.e., executables are on the nfs mount).

The issue is that client machines will suddenly enter a state where
they start spamming nfs traffic and become unusable. The nfs traffic
is apparently an infinite loop of the following:

-> PUTFH
<- NFS4_OK
-> READ
<- NFS4ERR_STALE_STATEID
-> RENEW
<- NFS4_OK

This repeats infinitely. The filehandle, clientid and stateid used are
always the same in all iterations of the loop. I have pcap captures I
can send, if this is useful (I'm afraid adding an attachment to my
initial message to the list will put me on the wrong side of the spam
filter).

When this starts happening, one process (the specific process is never
the same) on the client machine freezes completely. The same
executable cannot be run again -- it will similarly freeze. These
processes can be killed, but the spam traffic does not stop. The only
fix we've found so far is a reboot of the client machine. This issue
has persisted across multiple reboots of the server machine, and does
not appear to be triggered by any particular event on the server.
There is also no deterministic trigger on the client side, though I
think it seems to happen in periods of high traffic.

The server is nfs 4.0 (not 4.1). The server machine is Ubuntu 9.10
server (2.6.31-17-server); we've had this bug crop up on both Ubuntu
9.10 (2.6.31-17-generic) and Ubuntu 8.04 (not sure of kernel version)
desktops.

Now I know that Ubuntu has a patched kernel, and all else being equal
this should be a bug against Ubuntu. The reason I'm sending this to
this list is because I've spent the entire day reading the nfs4 spec
(rfc 3530) and reading the nfs client and server code in both Ubuntu
and latest-stable vanilla flavors, and
1) I'm confused and a little concerned about the way nfs4 works and I
would like to clarify my understanding, and
2) I hope I can help resolve the issue a little faster by working
directly with the devs, since I am also a dev (though not a kernel
dev).

I have not yet tried to understand how the client's stateid became
stale in the first place. The spec says that the client should be able
to deal with this happening at any time.

After reading the spec and the kernel code, it seems as though the
client code and server code have two different ideas about the
semantics of the RENEW operation.

The client code seems to respond to NFS4ERR_STALE_STATEID in almost
all cases by issuing a RENEW (via nfs4_schedule_state_recovery in
nfs/nfs4proc.c).

The server decides a stateid is stale if its si_boot time is new
enough (STALE_STATEID in nfsd/nfs4state.c). The server's RENEW
implementation (renew_client in nfsd/nfs4state.c) just resets its
"last active time" (nfs4client::cl_time, and the lru) to renew its
leases, which does not appear to reset the si_boot time of any
associated stateids. In fact, the only thing that seems to set
stateid_t::si_boot (thus freshening the stateid) are the OPEN and LOCK
operations, and friends.

Going by server implementation it looks like if client sees
STALE_STATEID, it must re-OPEN/LOCK a filehandle.

However, I can't tell from rfc 3530 whether this is correct. It says:

"The RENEW operation is used by the client to renew leases which it
currently holds at a server."

And that a lease is

"An interval of time defined by the server for which the client is
irrevocably granted a lock."

And a stateid:

"A 128-bit quantity returned by a server that uniquely defines the
open and locking state provided by the server for a specific open or
lock owner for a specific file."

I have not wrapped my head around the lock/lease/stateid trinity yet.
A strict interpretation points out that a stateid is "_open_ and
locking state", so therefore the only proper response to STALE_STATEID
is an OPEN. A loose interpretation says that they're all the same, and
that RENEW should refresh stateids.

It seems to me to be that the client was written with one assumption,
and the server with the other. The READ/RENEW loop seems like the
obvious outcome from that.

There's also a third valid perspective:

"The client must also employ the SETCLIENTID operation when it
receives a NFS4ERR_STALE_STATEID error using a stateid derived from
its current clientid."

As far as I can tell, the SETCLIENTID path only gets triggered on
NFS4ERR_STALE_CLIENTID.

I am sure I have missed some important things somewhere, because I
can't believe the implementation could be so divergent from the spec
as I think it is. However it's currently the only way I have to
explain the READ/RENEW loop bug that's taking down all the machines
where I work.

What have I missed?

Thanks very much in advance,

~Steve
J. Bruce Fields
2010-01-17 22:36:45 UTC
Permalink
Post by Steven Brudenell
Hello list,
I'm having a pretty nasty issue with nfs4. I work in a Linux
environment where a lot of software is run off of a central nfs4
server (i.e., executables are on the nfs mount).
The issue is that client machines will suddenly enter a state where
they start spamming nfs traffic and become unusable. The nfs traffic
-> PUTFH
<- NFS4_OK
-> READ
<- NFS4ERR_STALE_STATEID
-> RENEW
<- NFS4_OK
This repeats infinitely. The filehandle, clientid and stateid used are
always the same in all iterations of the loop. I have pcap captures I
can send, if this is useful (I'm afraid adding an attachment to my
initial message to the list will put me on the wrong side of the spam
filter).
Thanks for the report.

STALE_STATEID should mean the stateid is from a previous server boot,
but if that's so, then the clientid provided in the RENEW should also
be, and the RENEW should get a STALE_CLIENTID error. Probably a server
bug--I'll take another look at that code. Might be worth reverting
78155ed75f470710f2aecb3e75e3d97107ba8374.

--b.
Post by Steven Brudenell
When this starts happening, one process (the specific process is never
the same) on the client machine freezes completely. The same
executable cannot be run again -- it will similarly freeze. These
processes can be killed, but the spam traffic does not stop. The only
fix we've found so far is a reboot of the client machine. This issue
has persisted across multiple reboots of the server machine, and does
not appear to be triggered by any particular event on the server.
There is also no deterministic trigger on the client side, though I
think it seems to happen in periods of high traffic.
The server is nfs 4.0 (not 4.1). The server machine is Ubuntu 9.10
server (2.6.31-17-server); we've had this bug crop up on both Ubuntu
9.10 (2.6.31-17-generic) and Ubuntu 8.04 (not sure of kernel version)
desktops.
Now I know that Ubuntu has a patched kernel, and all else being equal
this should be a bug against Ubuntu. The reason I'm sending this to
this list is because I've spent the entire day reading the nfs4 spec
(rfc 3530) and reading the nfs client and server code in both Ubuntu
and latest-stable vanilla flavors, and
1) I'm confused and a little concerned about the way nfs4 works and I
would like to clarify my understanding, and
2) I hope I can help resolve the issue a little faster by working
directly with the devs, since I am also a dev (though not a kernel
dev).
I have not yet tried to understand how the client's stateid became
stale in the first place. The spec says that the client should be able
to deal with this happening at any time.
After reading the spec and the kernel code, it seems as though the
client code and server code have two different ideas about the
semantics of the RENEW operation.
The client code seems to respond to NFS4ERR_STALE_STATEID in almost
all cases by issuing a RENEW (via nfs4_schedule_state_recovery in
nfs/nfs4proc.c).
The server decides a stateid is stale if its si_boot time is new
enough (STALE_STATEID in nfsd/nfs4state.c). The server's RENEW
implementation (renew_client in nfsd/nfs4state.c) just resets its
"last active time" (nfs4client::cl_time, and the lru) to renew its
leases, which does not appear to reset the si_boot time of any
associated stateids. In fact, the only thing that seems to set
stateid_t::si_boot (thus freshening the stateid) are the OPEN and LOCK
operations, and friends.
Going by server implementation it looks like if client sees
STALE_STATEID, it must re-OPEN/LOCK a filehandle.
"The RENEW operation is used by the client to renew leases which it
currently holds at a server."
And that a lease is
"An interval of time defined by the server for which the client is
irrevocably granted a lock."
"A 128-bit quantity returned by a server that uniquely defines the
open and locking state provided by the server for a specific open or
lock owner for a specific file."
I have not wrapped my head around the lock/lease/stateid trinity yet.
A strict interpretation points out that a stateid is "_open_ and
locking state", so therefore the only proper response to STALE_STATEID
is an OPEN. A loose interpretation says that they're all the same, and
that RENEW should refresh stateids.
It seems to me to be that the client was written with one assumption,
and the server with the other. The READ/RENEW loop seems like the
obvious outcome from that.
"The client must also employ the SETCLIENTID operation when it
receives a NFS4ERR_STALE_STATEID error using a stateid derived from
its current clientid."
As far as I can tell, the SETCLIENTID path only gets triggered on
NFS4ERR_STALE_CLIENTID.
I am sure I have missed some important things somewhere, because I
can't believe the implementation could be so divergent from the spec
as I think it is. However it's currently the only way I have to
explain the READ/RENEW loop bug that's taking down all the machines
where I work.
What have I missed?
Thanks very much in advance,
~Steve
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
Steven Brudenell
2010-01-17 23:03:18 UTC
Permalink
Bruce,

Thanks for looking at this. It's causing a lot of pain for us right now.

I'm looking at 78155ed75f470710f2aecb3e75e3d97107ba8374 now. Offhand
I'm not sure that would fix the problem.

I managed to boil down my spec/code confusion to a simpler question:

Should RENEW freshen a client's stateids?

(If not, why does the kernel client code depend on it this way, and
how should a client otherwise respond to NFS4ERR_STALE_STATEID? If so,
why doesn't the kernel server code implement this?)

~Steve
Post by J. Bruce Fields
Post by Steven Brudenell
Hello list,
I'm having a pretty nasty issue with nfs4. I work in a Linux
environment where a lot of software is run off of a central nfs4
server (i.e., executables are on the nfs mount).
The issue is that client machines will suddenly enter a state where
they start spamming nfs traffic and become unusable. The nfs traffic
-> PUTFH
<- NFS4_OK
-> READ
<- NFS4ERR_STALE_STATEID
-> RENEW
<- NFS4_OK
This repeats infinitely. The filehandle, clientid and stateid used are
always the same in all iterations of the loop. I have pcap captures I
can send, if this is useful (I'm afraid adding an attachment to my
initial message to the list will put me on the wrong side of the spam
filter).
Thanks for the report.
STALE_STATEID should mean the stateid is from a previous server boot,
but if that's so, then the clientid provided in the RENEW should also
be, and the RENEW should get a STALE_CLIENTID error. ?Probably a server
bug--I'll take another look at that code. ?Might be worth reverting
78155ed75f470710f2aecb3e75e3d97107ba8374.
--b.
Post by Steven Brudenell
When this starts happening, one process (the specific process is never
the same) on the client machine freezes completely. The same
executable cannot be run again -- it will similarly freeze. These
processes can be killed, but the spam traffic does not stop. The only
fix we've found so far is a reboot of the client machine. This issue
has persisted across multiple reboots of the server machine, and does
not appear to be triggered by any particular event on the server.
There is also no deterministic trigger on the client side, though I
think it seems to happen in periods of high traffic.
The server is nfs 4.0 (not 4.1). The server machine is Ubuntu 9.10
server (2.6.31-17-server); we've had this bug crop up on both Ubuntu
9.10 (2.6.31-17-generic) and Ubuntu 8.04 (not sure of kernel version)
desktops.
Now I know that Ubuntu has a patched kernel, and all else being equal
this should be a bug against Ubuntu. The reason I'm sending this to
this list is because I've spent the entire day reading the nfs4 spec
(rfc 3530) and reading the nfs client and server code in both Ubuntu
and latest-stable vanilla flavors, and
1) I'm confused and a little concerned about the way nfs4 works and I
would like to clarify my understanding, and
2) I hope I can help resolve the issue a little faster by working
directly with the devs, since I am also a dev (though not a kernel
dev).
I have not yet tried to understand how the client's stateid became
stale in the first place. The spec says that the client should be able
to deal with this happening at any time.
After reading the spec and the kernel code, it seems as though the
client code and server code have two different ideas about the
semantics of the RENEW operation.
The client code seems to respond to NFS4ERR_STALE_STATEID in almost
all cases by issuing a RENEW (via nfs4_schedule_state_recovery in
nfs/nfs4proc.c).
The server decides a stateid is stale if its si_boot time is new
enough (STALE_STATEID in nfsd/nfs4state.c). The server's RENEW
implementation (renew_client in nfsd/nfs4state.c) just resets its
"last active time" (nfs4client::cl_time, and the lru) to renew its
leases, which does not appear to reset the si_boot time of any
associated stateids. In fact, the only thing that seems to set
stateid_t::si_boot (thus freshening the stateid) are the OPEN and LOCK
operations, and friends.
Going by server implementation it looks like if client sees
STALE_STATEID, it must re-OPEN/LOCK a filehandle.
"The RENEW operation is used by the client to renew leases which it
currently holds at a server."
And that a lease is
"An interval of time defined by the server for which the client is
irrevocably granted a lock."
"A 128-bit quantity returned by a server that uniquely defines the
open and locking state provided by the server for a specific open or
lock owner for a specific file."
I have not wrapped my head around the lock/lease/stateid trinity yet.
A strict interpretation points out that a stateid is "_open_ and
locking state", so therefore the only proper response to STALE_STATEID
is an OPEN. A loose interpretation says that they're all the same, and
that RENEW should refresh stateids.
It seems to me to be that the client was written with one assumption,
and the server with the other. The READ/RENEW loop seems like the
obvious outcome from that.
"The client must also employ the SETCLIENTID operation when it
receives a NFS4ERR_STALE_STATEID error using a stateid derived from
its current clientid."
As far as I can tell, the SETCLIENTID path only gets triggered on
NFS4ERR_STALE_CLIENTID.
I am sure I have missed some important things somewhere, because I
can't believe the implementation could be so divergent from the spec
as I think it is. However it's currently the only way I have to
explain the READ/RENEW loop bug that's taking down all the machines
where I work.
What have I missed?
Thanks very much in advance,
~Steve
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
J. Bruce Fields
2010-01-18 21:42:50 UTC
Permalink
Post by Steven Brudenell
Bruce,
Thanks for looking at this. It's causing a lot of pain for us right now.
I'm looking at 78155ed75f470710f2aecb3e75e3d97107ba8374 now. Offhand
I'm not sure that would fix the problem.
Should RENEW freshen a client's stateids?
A stateid is "stale" if the server has rebooted since it was given out.
There's no way to make it not stale after that.
Post by Steven Brudenell
(If not, why does the kernel client code depend on it this way, and
how should a client otherwise respond to NFS4ERR_STALE_STATEID? If so,
why doesn't the kernel server code implement this?)
The RENEW isn't an attempt to "freshen" anything; it's just attempting
to confirm what the situation is with the server. (As far as I can tell
the RENEW is redundant in this case; once one of the stateid's is
declared "stale", they all should be; so I think the RENEW is there just
because some other state-recovery cases need it and it doesn't hurt.)

In any case: I'm not seeing an obvious cause. Could you send me a
network trace? Anything just showing one complete loop would be fine,
as long as it includes the entire contents of each packet, not just the
start or a summary. You can send it privately if you want.

--b.
Post by Steven Brudenell
~Steve
Post by J. Bruce Fields
Post by Steven Brudenell
Hello list,
I'm having a pretty nasty issue with nfs4. I work in a Linux
environment where a lot of software is run off of a central nfs4
server (i.e., executables are on the nfs mount).
The issue is that client machines will suddenly enter a state where
they start spamming nfs traffic and become unusable. The nfs traffic
-> PUTFH
<- NFS4_OK
-> READ
<- NFS4ERR_STALE_STATEID
-> RENEW
<- NFS4_OK
This repeats infinitely. The filehandle, clientid and stateid used are
always the same in all iterations of the loop. I have pcap captures I
can send, if this is useful (I'm afraid adding an attachment to my
initial message to the list will put me on the wrong side of the spam
filter).
Thanks for the report.
STALE_STATEID should mean the stateid is from a previous server boot,
but if that's so, then the clientid provided in the RENEW should also
be, and the RENEW should get a STALE_CLIENTID error. ?Probably a server
bug--I'll take another look at that code. ?Might be worth reverting
78155ed75f470710f2aecb3e75e3d97107ba8374.
--b.
Post by Steven Brudenell
When this starts happening, one process (the specific process is never
the same) on the client machine freezes completely. The same
executable cannot be run again -- it will similarly freeze. These
processes can be killed, but the spam traffic does not stop. The only
fix we've found so far is a reboot of the client machine. This issue
has persisted across multiple reboots of the server machine, and does
not appear to be triggered by any particular event on the server.
There is also no deterministic trigger on the client side, though I
think it seems to happen in periods of high traffic.
The server is nfs 4.0 (not 4.1). The server machine is Ubuntu 9.10
server (2.6.31-17-server); we've had this bug crop up on both Ubuntu
9.10 (2.6.31-17-generic) and Ubuntu 8.04 (not sure of kernel version)
desktops.
Now I know that Ubuntu has a patched kernel, and all else being equal
this should be a bug against Ubuntu. The reason I'm sending this to
this list is because I've spent the entire day reading the nfs4 spec
(rfc 3530) and reading the nfs client and server code in both Ubuntu
and latest-stable vanilla flavors, and
1) I'm confused and a little concerned about the way nfs4 works and I
would like to clarify my understanding, and
2) I hope I can help resolve the issue a little faster by working
directly with the devs, since I am also a dev (though not a kernel
dev).
I have not yet tried to understand how the client's stateid became
stale in the first place. The spec says that the client should be able
to deal with this happening at any time.
After reading the spec and the kernel code, it seems as though the
client code and server code have two different ideas about the
semantics of the RENEW operation.
The client code seems to respond to NFS4ERR_STALE_STATEID in almost
all cases by issuing a RENEW (via nfs4_schedule_state_recovery in
nfs/nfs4proc.c).
The server decides a stateid is stale if its si_boot time is new
enough (STALE_STATEID in nfsd/nfs4state.c). The server's RENEW
implementation (renew_client in nfsd/nfs4state.c) just resets its
"last active time" (nfs4client::cl_time, and the lru) to renew its
leases, which does not appear to reset the si_boot time of any
associated stateids. In fact, the only thing that seems to set
stateid_t::si_boot (thus freshening the stateid) are the OPEN and LOCK
operations, and friends.
Going by server implementation it looks like if client sees
STALE_STATEID, it must re-OPEN/LOCK a filehandle.
"The RENEW operation is used by the client to renew leases which it
currently holds at a server."
And that a lease is
"An interval of time defined by the server for which the client is
irrevocably granted a lock."
"A 128-bit quantity returned by a server that uniquely defines the
open and locking state provided by the server for a specific open or
lock owner for a specific file."
I have not wrapped my head around the lock/lease/stateid trinity yet.
A strict interpretation points out that a stateid is "_open_ and
locking state", so therefore the only proper response to STALE_STATEID
is an OPEN. A loose interpretation says that they're all the same, and
that RENEW should refresh stateids.
It seems to me to be that the client was written with one assumption,
and the server with the other. The READ/RENEW loop seems like the
obvious outcome from that.
"The client must also employ the SETCLIENTID operation when it
receives a NFS4ERR_STALE_STATEID error using a stateid derived from
its current clientid."
As far as I can tell, the SETCLIENTID path only gets triggered on
NFS4ERR_STALE_CLIENTID.
I am sure I have missed some important things somewhere, because I
can't believe the implementation could be so divergent from the spec
as I think it is. However it's currently the only way I have to
explain the READ/RENEW loop bug that's taking down all the machines
where I work.
What have I missed?
Thanks very much in advance,
~Steve
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
J. Bruce Fields
2010-01-18 23:59:40 UTC
Permalink
Post by J. Bruce Fields
In any case: I'm not seeing an obvious cause. Could you send me a
network trace? Anything just showing one complete loop would be fine,
as long as it includes the entire contents of each packet, not just the
start or a summary. You can send it privately if you want.
OK, got it, thanks.

The current linux server implementation is keeping the time of the
stateid's creation in the first four bytes of the stateid and the
server's startup time in the first four bytes of the clientid.

The stateid in the read has (after swapping bytes), 4b4f9021, and the
clientid 4b50aca7. Which would have the stateid created 72838 seconds
(just over 20 hours) before the nfs server started. So time went
backwards somewhere.

Maybe there's something else we could rely on here instead of time. Or
maybe we should reconsider 02cb2858dbfe216bd4a91c4afb4e259ef3b9e151.

I'd be curious to hear if you could figure out a cause for that time
jump, to help decide whether this is a problem that's likely to be
common.

--b.
Steven Brudenell
2010-01-19 00:10:18 UTC
Permalink
Post by J. Bruce Fields
In any case: I'm not seeing an obvious cause. ?Could you send me a
network trace? ?Anything just showing one complete loop would be fine,
as long as it includes the entire contents of each packet, not just the
start or a summary. ?You can send it privately if you want.
OK, got it, thanks.
The current linux server implementation is keeping the time of the
stateid's creation in the first four bytes of the stateid and the
server's startup time in the first four bytes of the clientid.
The stateid in the read has (after swapping bytes), 4b4f9021, and the
clientid 4b50aca7. ?Which would have the stateid created 72838 seconds
(just over 20 hours) before the nfs server started. ?So time went
backwards somewhere.
Maybe there's something else we could rely on here instead of time. ?Or
maybe we should reconsider 02cb2858dbfe216bd4a91c4afb4e259ef3b9e151.
I'd be curious to hear if you could figure out a cause for that time
jump, to help decide whether this is a problem that's likely to be
common.
I'm beyond clueless on this one. I talked to our admin, apparently the
server doesn't even have ntp scripts installed. My only clue is that
clients only get stuck in the loop very infrequently (as opposed to
all at once), and the time on the server is correct-ish now, so I
guess the time madness is happening in fairly narrow windows.

That aside, regardless of how common it is, it's certainly bad if a
wrong system time on the server sends clients into an unrecoverable
frenzy.

While I understand your desire to fix the root cause, I repeat my
request that the client code do state recovery on STALE_STATEID, the
same as is currently done on STALE_CLIENTID. It seems more correct for
the client code to do this (especially according to the RFC), and it
would probably take less time to code than a refactoring on the server
(or even deciding how to refactor -- it's not clear how the server
would do lease expiration without relying on time).
Post by J. Bruce Fields
--b.
David Brodbeck
2010-01-19 21:41:15 UTC
Permalink
Post by J. Bruce Fields
I'd be curious to hear if you could figure out a cause for that time
jump, to help decide whether this is a problem that's likely to be
common.
I haven't seen this particular problem, but this kind of time jump is pretty common if a server reboots with an incorrect hardware clock, then either runs ntpdate or runs ntpd with the "sanity check" disabled.
--
David Brodbeck
System Administrator, Linguistics
University of Washington
Steven Brudenell
2010-01-20 02:28:45 UTC
Permalink
To clarify, since I can't remember if I said this in private to Bruce
or to the list:

There isn't any trigger for this bug such as a rebooting server. Both
client and server are humming along fine, when suddenly the client
ends up with a stateid twenty hours older than the clientid.
Post by David Brodbeck
Post by J. Bruce Fields
I'd be curious to hear if you could figure out a cause for that time
jump, to help decide whether this is a problem that's likely to be
common.
I haven't seen this particular problem, but this kind of time jump is pretty common if a server reboots with an incorrect hardware clock, then either runs ntpdate or runs ntpd with the "sanity check" disabled.
--
David Brodbeck
System Administrator, Linguistics
University of Washington
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
J. Bruce Fields
2010-01-21 21:56:31 UTC
Permalink
Post by Steven Brudenell
To clarify, since I can't remember if I said this in private to Bruce
There isn't any trigger for this bug such as a rebooting server. Both
client and server are humming along fine, when suddenly the client
ends up with a stateid twenty hours older than the clientid.
Well, then maybe it is possible there's some client bug that's causing
it to use the wrong stateid. I'm not sure how to look for that.

--b.
J. Bruce Fields
2010-01-21 22:01:32 UTC
Permalink
Post by J. Bruce Fields
Post by Steven Brudenell
To clarify, since I can't remember if I said this in private to Bruce
There isn't any trigger for this bug such as a rebooting server. Both
client and server are humming along fine, when suddenly the client
ends up with a stateid twenty hours older than the clientid.
Well, then maybe it is possible there's some client bug that's causing
it to use the wrong stateid. I'm not sure how to look for that.
Looking through git, there has been a fair amount of change to the
client recovery code; might be worth trying the latest upstream kernel
and seeing if the problem's still reproduceable. Also, perhaps

a78cb57a106fceeba26da2907db9d8886700e1dc

would at least prevent the infinite loop. (I'm not sure if the code's
getting to this point in your example.)

--b.
Trond Myklebust
2010-01-21 22:12:48 UTC
Permalink
Post by J. Bruce Fields
Post by Steven Brudenell
To clarify, since I can't remember if I said this in private to Bruce
There isn't any trigger for this bug such as a rebooting server. Both
client and server are humming along fine, when suddenly the client
ends up with a stateid twenty hours older than the clientid.
Well, then maybe it is possible there's some client bug that's causing
it to use the wrong stateid. I'm not sure how to look for that.
There was a time when the renew daemon was being turned off every time
you unmounted a filesystem from the same server. That bug was fixed by
commit 3050141bae57984dd660e6861632ccf9b8bca77e (NFSv4: Kill
nfs4_renewd_prepare_shutdown()).

However I can see that even in today's tree, we don't guarantee recovery
from the NFS4ERR_STALE_STATEID error. I'm therefore thinking of
committing something like the appended patch to ensure that we do...

Cheers
Trond
---------------------------------------------------------------------------------------
NFS: Ensure that we handle NFS4ERR_STALE_STATEID correctly

From: Trond Myklebust <Trond.Myklebust at netapp.com>

Even if the server is crazy, we should be able to mark the stateid as being
bad, to ensure it gets recovered.

Signed-off-by: Trond Myklebust <Trond.Myklebust at netapp.com>
---

fs/nfs/nfs4_fs.h | 1 +
fs/nfs/nfs4proc.c | 38 +++++++++++++++++++++++++-------------
fs/nfs/nfs4state.c | 2 +-
3 files changed, 27 insertions(+), 14 deletions(-)


diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index ea2f41b..0c6fda3 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -278,6 +278,7 @@ extern void nfs4_state_set_mode_locked(struct nfs4_state *, fmode_t);
extern void nfs4_schedule_state_recovery(struct nfs_client *);
extern void nfs4_schedule_state_manager(struct nfs_client *);
extern int nfs4_state_mark_reclaim_nograce(struct nfs_client *clp, struct nfs4_state *state);
+extern int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp, struct nfs4_state *state);
extern void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags);
extern void nfs4_put_lock_state(struct nfs4_lock_state *lsp);
extern int nfs4_set_lock_state(struct nfs4_state *state, struct file_lock *fl);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index afbfe67..dca0ebc 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -249,14 +249,14 @@ static int nfs4_handle_exception(const struct nfs_server *server, int errorcode,
if (state == NULL)
break;
nfs4_state_mark_reclaim_nograce(clp, state);
- case -NFS4ERR_STALE_CLIENTID:
+ goto do_state_recovery;
case -NFS4ERR_STALE_STATEID:
+ if (state == NULL)
+ break;
+ nfs4_state_mark_reclaim_reboot(clp, state);
+ case -NFS4ERR_STALE_CLIENTID:
case -NFS4ERR_EXPIRED:
- nfs4_schedule_state_recovery(clp);
- ret = nfs4_wait_clnt_recover(clp);
- if (ret == 0)
- exception->retry = 1;
- break;
+ goto do_state_recovery;
#if defined(CONFIG_NFS_V4_1)
case -NFS4ERR_BADSESSION:
case -NFS4ERR_BADSLOT:
@@ -289,6 +289,12 @@ static int nfs4_handle_exception(const struct nfs_server *server, int errorcode,
}
/* We failed to handle the error */
return nfs4_map_errors(ret);
+do_state_recovery:
+ nfs4_schedule_state_recovery(clp);
+ ret = nfs4_wait_clnt_recover(clp);
+ if (ret == 0)
+ exception->retry = 1;
+ return ret;
}


@@ -3420,15 +3426,14 @@ _nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
if (state == NULL)
break;
nfs4_state_mark_reclaim_nograce(clp, state);
- case -NFS4ERR_STALE_CLIENTID:
+ goto do_state_recovery;
case -NFS4ERR_STALE_STATEID:
+ if (state == NULL)
+ break;
+ nfs4_state_mark_reclaim_reboot(clp, state);
+ case -NFS4ERR_STALE_CLIENTID:
case -NFS4ERR_EXPIRED:
- rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
- nfs4_schedule_state_recovery(clp);
- if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
- rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
- task->tk_status = 0;
- return -EAGAIN;
+ goto do_state_recovery;
#if defined(CONFIG_NFS_V4_1)
case -NFS4ERR_BADSESSION:
case -NFS4ERR_BADSLOT:
@@ -3456,6 +3461,13 @@ _nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
}
task->tk_status = nfs4_map_errors(task->tk_status);
return 0;
+do_state_recovery:
+ rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
+ nfs4_schedule_state_recovery(clp);
+ if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
+ rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
+ task->tk_status = 0;
+ return -EAGAIN;
}

static int
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 6d263ed..c1e2733 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -901,7 +901,7 @@ void nfs4_schedule_state_recovery(struct nfs_client *clp)
nfs4_schedule_state_manager(clp);
}

-static int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp, struct nfs4_state *state)
+int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp, struct nfs4_state *state)
{

set_bit(NFS_STATE_RECLAIM_REBOOT, &state->flags);
J. Bruce Fields
2010-01-21 22:20:55 UTC
Permalink
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
To clarify, since I can't remember if I said this in private to Bruce
There isn't any trigger for this bug such as a rebooting server. Both
client and server are humming along fine, when suddenly the client
ends up with a stateid twenty hours older than the clientid.
Well, then maybe it is possible there's some client bug that's causing
it to use the wrong stateid. I'm not sure how to look for that.
There was a time when the renew daemon was being turned off every time
you unmounted a filesystem from the same server. That bug was fixed by
commit 3050141bae57984dd660e6861632ccf9b8bca77e (NFSv4: Kill
nfs4_renewd_prepare_shutdown()).
However I can see that even in today's tree, we don't guarantee recovery
from the NFS4ERR_STALE_STATEID error. I'm therefore thinking of
committing something like the appended patch to ensure that we do...
Makes sense. The one disadvantage from my point of view is that it
might cover up a server problem that we'd rather know about. But
there's not much to be done about that. (Will the application wind up
getting an IO error in this case, or does the client try to get a new
open?)

--b.
Post by Trond Myklebust
Cheers
Trond
---------------------------------------------------------------------------------------
NFS: Ensure that we handle NFS4ERR_STALE_STATEID correctly
From: Trond Myklebust <Trond.Myklebust at netapp.com>
Even if the server is crazy, we should be able to mark the stateid as being
bad, to ensure it gets recovered.
Signed-off-by: Trond Myklebust <Trond.Myklebust at netapp.com>
---
fs/nfs/nfs4_fs.h | 1 +
fs/nfs/nfs4proc.c | 38 +++++++++++++++++++++++++-------------
fs/nfs/nfs4state.c | 2 +-
3 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h
index ea2f41b..0c6fda3 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -278,6 +278,7 @@ extern void nfs4_state_set_mode_locked(struct nfs4_state *, fmode_t);
extern void nfs4_schedule_state_recovery(struct nfs_client *);
extern void nfs4_schedule_state_manager(struct nfs_client *);
extern int nfs4_state_mark_reclaim_nograce(struct nfs_client *clp, struct nfs4_state *state);
+extern int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp, struct nfs4_state *state);
extern void nfs41_handle_sequence_flag_errors(struct nfs_client *clp, u32 flags);
extern void nfs4_put_lock_state(struct nfs4_lock_state *lsp);
extern int nfs4_set_lock_state(struct nfs4_state *state, struct file_lock *fl);
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index afbfe67..dca0ebc 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -249,14 +249,14 @@ static int nfs4_handle_exception(const struct nfs_server *server, int errorcode,
if (state == NULL)
break;
nfs4_state_mark_reclaim_nograce(clp, state);
+ goto do_state_recovery;
+ if (state == NULL)
+ break;
+ nfs4_state_mark_reclaim_reboot(clp, state);
- nfs4_schedule_state_recovery(clp);
- ret = nfs4_wait_clnt_recover(clp);
- if (ret == 0)
- exception->retry = 1;
- break;
+ goto do_state_recovery;
#if defined(CONFIG_NFS_V4_1)
@@ -289,6 +289,12 @@ static int nfs4_handle_exception(const struct nfs_server *server, int errorcode,
}
/* We failed to handle the error */
return nfs4_map_errors(ret);
+ nfs4_schedule_state_recovery(clp);
+ ret = nfs4_wait_clnt_recover(clp);
+ if (ret == 0)
+ exception->retry = 1;
+ return ret;
}
@@ -3420,15 +3426,14 @@ _nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
if (state == NULL)
break;
nfs4_state_mark_reclaim_nograce(clp, state);
+ goto do_state_recovery;
+ if (state == NULL)
+ break;
+ nfs4_state_mark_reclaim_reboot(clp, state);
- rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
- nfs4_schedule_state_recovery(clp);
- if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
- rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
- task->tk_status = 0;
- return -EAGAIN;
+ goto do_state_recovery;
#if defined(CONFIG_NFS_V4_1)
@@ -3456,6 +3461,13 @@ _nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
}
task->tk_status = nfs4_map_errors(task->tk_status);
return 0;
+ rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
+ nfs4_schedule_state_recovery(clp);
+ if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
+ rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
+ task->tk_status = 0;
+ return -EAGAIN;
}
static int
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 6d263ed..c1e2733 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -901,7 +901,7 @@ void nfs4_schedule_state_recovery(struct nfs_client *clp)
nfs4_schedule_state_manager(clp);
}
-static int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp, struct nfs4_state *state)
+int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp, struct nfs4_state *state)
{
set_bit(NFS_STATE_RECLAIM_REBOOT, &state->flags);
Muntz, Daniel
2010-01-21 22:40:46 UTC
Permalink
-----Original Message-----
From: J. Bruce Fields [mailto:bfields at fieldses.org]
Sent: Thursday, January 21, 2010 2:21 PM
To: Myklebust, Trond
Cc: nfsv4 at linux-nfs.org
Subject: Re: READ/RENEW loop bug
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
To clarify, since I can't remember if I said this in private to
There isn't any trigger for this bug such as a
rebooting server.
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
Both client and server are humming along fine, when
suddenly the
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
client ends up with a stateid twenty hours older than
the clientid.
Post by Trond Myklebust
Post by J. Bruce Fields
Well, then maybe it is possible there's some client bug that's
causing it to use the wrong stateid. I'm not sure how to
look for that.
Post by Trond Myklebust
There was a time when the renew daemon was being turned off
every time
Post by Trond Myklebust
you unmounted a filesystem from the same server. That bug
was fixed by
Post by Trond Myklebust
commit 3050141bae57984dd660e6861632ccf9b8bca77e (NFSv4: Kill
nfs4_renewd_prepare_shutdown()).
However I can see that even in today's tree, we don't guarantee
recovery from the NFS4ERR_STALE_STATEID error. I'm
therefore thinking
Post by Trond Myklebust
of committing something like the appended patch to ensure
that we do...
Makes sense. The one disadvantage from my point of view is
that it might cover up a server problem that we'd rather know
about. But there's not much to be done about that. (Will
the application wind up getting an IO error in this case, or
does the client try to get a new
open?)
Yeah, I was wondering if there might be a server error here that would
get masked. When I get some time, there's an error I can reproduce
occasionally that _seems_ like the server is "forgetting" a stateid.
There's not much to see in traces when this happens--the client gets a
stateid and the server (sometimes quickly, sometimes after some time),
starts returning NFS4ERR_BAD_STATEID, and eventually starts returning
NFS4ERR_EXPIRED. The client (intentionally) is stubbornly using the
same stateid because there's no obvious reason that the stateid should
be invalid.

-Dan
--b.
Post by Trond Myklebust
Cheers
Trond
----------------------------------------------------------------------
Post by Trond Myklebust
-----------------
NFS: Ensure that we handle NFS4ERR_STALE_STATEID correctly
From: Trond Myklebust <Trond.Myklebust at netapp.com>
Even if the server is crazy, we should be able to mark the
stateid as
Post by Trond Myklebust
being bad, to ensure it gets recovered.
Signed-off-by: Trond Myklebust <Trond.Myklebust at netapp.com>
---
fs/nfs/nfs4_fs.h | 1 +
fs/nfs/nfs4proc.c | 38 +++++++++++++++++++++++++-------------
fs/nfs/nfs4state.c | 2 +-
3 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h index
ea2f41b..0c6fda3 100644
--- a/fs/nfs/nfs4_fs.h
+++ b/fs/nfs/nfs4_fs.h
@@ -278,6 +278,7 @@ extern void nfs4_state_set_mode_locked(struct
nfs4_state *, fmode_t); extern void
nfs4_schedule_state_recovery(struct nfs_client *); extern void
nfs4_schedule_state_manager(struct nfs_client *); extern int
nfs4_state_mark_reclaim_nograce(struct nfs_client *clp, struct
nfs4_state *state);
+extern int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp,
+struct nfs4_state *state);
extern void nfs41_handle_sequence_flag_errors(struct
nfs_client *clp,
Post by Trond Myklebust
u32 flags); extern void nfs4_put_lock_state(struct nfs4_lock_state
*lsp); extern int nfs4_set_lock_state(struct nfs4_state *state,
struct file_lock *fl); diff --git a/fs/nfs/nfs4proc.c
b/fs/nfs/nfs4proc.c index afbfe67..dca0ebc 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -249,14 +249,14 @@ static int
nfs4_handle_exception(const struct nfs_server *server, int errorcode,
Post by Trond Myklebust
if (state == NULL)
break;
nfs4_state_mark_reclaim_nograce(clp, state);
+ goto do_state_recovery;
+ if (state == NULL)
+ break;
+ nfs4_state_mark_reclaim_reboot(clp, state);
- nfs4_schedule_state_recovery(clp);
- ret = nfs4_wait_clnt_recover(clp);
- if (ret == 0)
- exception->retry = 1;
- break;
+ goto do_state_recovery;
#if defined(CONFIG_NFS_V4_1)
@@ -289,6 +289,12 @@ static int nfs4_handle_exception(const
struct nfs_server *server, int errorcode,
Post by Trond Myklebust
}
/* We failed to handle the error */
return nfs4_map_errors(ret);
+ nfs4_schedule_state_recovery(clp);
+ ret = nfs4_wait_clnt_recover(clp);
+ if (ret == 0)
+ exception->retry = 1;
+ return ret;
}
@@ -3420,15 +3426,14 @@ _nfs4_async_handle_error(struct
rpc_task *task, const struct nfs_server *server,
Post by Trond Myklebust
if (state == NULL)
break;
nfs4_state_mark_reclaim_nograce(clp, state);
+ goto do_state_recovery;
+ if (state == NULL)
+ break;
+ nfs4_state_mark_reclaim_reboot(clp, state);
- rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
- nfs4_schedule_state_recovery(clp);
- if (test_bit(NFS4CLNT_MANAGER_RUNNING,
&clp->cl_state) == 0)
Post by Trond Myklebust
-
rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
Post by Trond Myklebust
- task->tk_status = 0;
- return -EAGAIN;
+ goto do_state_recovery;
#if defined(CONFIG_NFS_V4_1)
@@ -3456,6 +3461,13 @@ _nfs4_async_handle_error(struct
rpc_task *task, const struct nfs_server *server,
Post by Trond Myklebust
}
task->tk_status = nfs4_map_errors(task->tk_status);
return 0;
+ rpc_sleep_on(&clp->cl_rpcwaitq, task, NULL);
+ nfs4_schedule_state_recovery(clp);
+ if (test_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state) == 0)
+ rpc_wake_up_queued_task(&clp->cl_rpcwaitq, task);
+ task->tk_status = 0;
+ return -EAGAIN;
}
static int
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index
6d263ed..c1e2733 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -901,7 +901,7 @@ void
nfs4_schedule_state_recovery(struct nfs_client *clp)
Post by Trond Myklebust
nfs4_schedule_state_manager(clp);
}
-static int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp,
struct nfs4_state *state)
+int nfs4_state_mark_reclaim_reboot(struct nfs_client *clp, struct
+nfs4_state *state)
{
set_bit(NFS_STATE_RECLAIM_REBOOT, &state->flags);
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
J. Bruce Fields
2010-01-21 22:58:16 UTC
Permalink
Post by Muntz, Daniel
-----Original Message-----
From: J. Bruce Fields [mailto:bfields at fieldses.org]
Sent: Thursday, January 21, 2010 2:21 PM
To: Myklebust, Trond
Cc: nfsv4 at linux-nfs.org
Subject: Re: READ/RENEW loop bug
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
To clarify, since I can't remember if I said this in private to
There isn't any trigger for this bug such as a
rebooting server.
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
Both client and server are humming along fine, when
suddenly the
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
client ends up with a stateid twenty hours older than
the clientid.
Post by Trond Myklebust
Post by J. Bruce Fields
Well, then maybe it is possible there's some client bug that's
causing it to use the wrong stateid. I'm not sure how to
look for that.
Post by Trond Myklebust
There was a time when the renew daemon was being turned off
every time
Post by Trond Myklebust
you unmounted a filesystem from the same server. That bug
was fixed by
Post by Trond Myklebust
commit 3050141bae57984dd660e6861632ccf9b8bca77e (NFSv4: Kill
nfs4_renewd_prepare_shutdown()).
However I can see that even in today's tree, we don't guarantee
recovery from the NFS4ERR_STALE_STATEID error. I'm
therefore thinking
Post by Trond Myklebust
of committing something like the appended patch to ensure
that we do...
Makes sense. The one disadvantage from my point of view is
that it might cover up a server problem that we'd rather know
about. But there's not much to be done about that. (Will
the application wind up getting an IO error in this case, or
does the client try to get a new
open?)
Yeah, I was wondering if there might be a server error here that would
get masked. When I get some time, there's an error I can reproduce
occasionally that _seems_ like the server is "forgetting" a stateid.
There's not much to see in traces when this happens--the client gets a
stateid and the server (sometimes quickly, sometimes after some time),
starts returning NFS4ERR_BAD_STATEID, and eventually starts returning
NFS4ERR_EXPIRED. The client (intentionally) is stubbornly using the
same stateid because there's no obvious reason that the stateid should
be invalid.
Hm. Any more data you can collect from one of those incidents would be
helpful, of course.

--b.
Trond Myklebust
2010-01-21 23:26:12 UTC
Permalink
Post by Muntz, Daniel
-----Original Message-----
From: J. Bruce Fields [mailto:bfields at fieldses.org]
Sent: Thursday, January 21, 2010 2:21 PM
To: Myklebust, Trond
Cc: nfsv4 at linux-nfs.org
Subject: Re: READ/RENEW loop bug
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
To clarify, since I can't remember if I said this in private to
There isn't any trigger for this bug such as a
rebooting server.
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
Both client and server are humming along fine, when
suddenly the
Post by Trond Myklebust
Post by J. Bruce Fields
Post by Steven Brudenell
client ends up with a stateid twenty hours older than
the clientid.
Post by Trond Myklebust
Post by J. Bruce Fields
Well, then maybe it is possible there's some client bug that's
causing it to use the wrong stateid. I'm not sure how to
look for that.
Post by Trond Myklebust
There was a time when the renew daemon was being turned off
every time
Post by Trond Myklebust
you unmounted a filesystem from the same server. That bug
was fixed by
Post by Trond Myklebust
commit 3050141bae57984dd660e6861632ccf9b8bca77e (NFSv4: Kill
nfs4_renewd_prepare_shutdown()).
However I can see that even in today's tree, we don't guarantee
recovery from the NFS4ERR_STALE_STATEID error. I'm
therefore thinking
Post by Trond Myklebust
of committing something like the appended patch to ensure
that we do...
Makes sense. The one disadvantage from my point of view is
that it might cover up a server problem that we'd rather know
about. But there's not much to be done about that. (Will
the application wind up getting an IO error in this case, or
does the client try to get a new
open?)
Yeah, I was wondering if there might be a server error here that would
get masked. When I get some time, there's an error I can reproduce
occasionally that _seems_ like the server is "forgetting" a stateid.
There's not much to see in traces when this happens--the client gets a
stateid and the server (sometimes quickly, sometimes after some time),
starts returning NFS4ERR_BAD_STATEID, and eventually starts returning
NFS4ERR_EXPIRED. The client (intentionally) is stubbornly using the
same stateid because there's no obvious reason that the stateid should
be invalid.
The handling of current_fileid appears to be buggy. There is no
protection against an overflow situation, so current_fileid == 0 is not
excluded. That would definitely give rise to "interesting" consequences.

I'm not sure about whether or not overflows of current_ownerid and
current_delegid are as worrisome, but there appears to be no guarantee
against collisions...

Trond
J. Bruce Fields
2010-01-27 18:47:37 UTC
Permalink
Post by Trond Myklebust
The handling of current_fileid appears to be buggy. There is no
protection against an overflow situation, so current_fileid == 0 is not
excluded. That would definitely give rise to "interesting" consequences.
Thanks to the double meaning of si_fileid == 0 as "this is for a
delegation"--yes.
Post by Trond Myklebust
I'm not sure about whether or not overflows of current_ownerid and
current_delegid are as worrisome, but there appears to be no guarantee
against collisions...
Yeah.

Maybe we should make the opaque part out of a boot time and a single
64-bit counter. I don't see why we need the separate fileid and
stateownerid.

--b.
Steven Brudenell
2010-02-01 20:04:09 UTC
Permalink
Hey folks,

Just wondering if there has been any progress on this issue as yet. I
don't think I've gone two days without rebooting my client machine in
a while. :(

~Steve
Post by J. Bruce Fields
Post by Trond Myklebust
The handling of current_fileid appears to be buggy. There is no
protection against an overflow situation, so current_fileid == 0 is not
excluded. That would definitely give rise to "interesting" consequences.
Thanks to the double meaning of si_fileid == 0 as "this is for a
delegation"--yes.
Post by Trond Myklebust
I'm not sure about whether or not overflows of current_ownerid and
current_delegid are as worrisome, but there appears to be no guarantee
against collisions...
Yeah.
Maybe we should make the opaque part out of a boot time and a single
64-bit counter. ?I don't see why we need the separate fileid and
stateownerid.
--b.
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
Steven Brudenell
2010-02-26 16:47:43 UTC
Permalink
Ping again (because my client machine crashed again). Did any change
ever get committed for this?

~Steve

On Mon, Feb 1, 2010 at 3:04 PM, Steven Brudenell
Post by Steven Brudenell
Hey folks,
Just wondering if there has been any progress on this issue as yet. ?I
don't think I've gone two days without rebooting my client machine in
a while. :(
~Steve
Post by J. Bruce Fields
Post by Trond Myklebust
The handling of current_fileid appears to be buggy. There is no
protection against an overflow situation, so current_fileid == 0 is not
excluded. That would definitely give rise to "interesting" consequences.
Thanks to the double meaning of si_fileid == 0 as "this is for a
delegation"--yes.
Post by Trond Myklebust
I'm not sure about whether or not overflows of current_ownerid and
current_delegid are as worrisome, but there appears to be no guarantee
against collisions...
Yeah.
Maybe we should make the opaque part out of a boot time and a single
64-bit counter. ?I don't see why we need the separate fileid and
stateownerid.
--b.
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
J. Bruce Fields
2010-02-26 23:09:48 UTC
Permalink
Post by Steven Brudenell
Ping again (because my client machine crashed again). Did any change
ever get committed for this?
Unfortunately I think we still haven't determined a cause. Have you
found out anything more about the crashes?

--b.
Post by Steven Brudenell
~Steve
On Mon, Feb 1, 2010 at 3:04 PM, Steven Brudenell
Post by Steven Brudenell
Hey folks,
Just wondering if there has been any progress on this issue as yet. ?I
don't think I've gone two days without rebooting my client machine in
a while. :(
~Steve
Post by J. Bruce Fields
Post by Trond Myklebust
The handling of current_fileid appears to be buggy. There is no
protection against an overflow situation, so current_fileid == 0 is not
excluded. That would definitely give rise to "interesting" consequences.
Thanks to the double meaning of si_fileid == 0 as "this is for a
delegation"--yes.
Post by Trond Myklebust
I'm not sure about whether or not overflows of current_ownerid and
current_delegid are as worrisome, but there appears to be no guarantee
against collisions...
Yeah.
Maybe we should make the opaque part out of a boot time and a single
64-bit counter. ?I don't see why we need the separate fileid and
stateownerid.
--b.
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
Steven Brudenell
2010-02-26 23:18:41 UTC
Permalink
Nothing at all.

I thought we had determined the proximal cause, since you verified
that the stateid was created 20 hours earlier than the clientid.

I still can't explain why our nfs server keeps going back in time.
However, a sysadmin playing with the clock would cause this.

I'll be happy to debug this further, but I'll need some guidance. Is
there some kind of hook I can set to log whenever the system time
changes? Is there anything else I can do to debug the nfs code?

~Steve
Post by Steven Brudenell
Ping again (because my client machine crashed again). Did any change
ever get committed for this?
Unfortunately I think we still haven't determined a cause. ?Have you
found out anything more about the crashes?
--b.
Post by Steven Brudenell
~Steve
On Mon, Feb 1, 2010 at 3:04 PM, Steven Brudenell
Post by Steven Brudenell
Hey folks,
Just wondering if there has been any progress on this issue as yet. ?I
don't think I've gone two days without rebooting my client machine in
a while. :(
~Steve
Post by J. Bruce Fields
Post by Trond Myklebust
The handling of current_fileid appears to be buggy. There is no
protection against an overflow situation, so current_fileid == 0 is not
excluded. That would definitely give rise to "interesting" consequences.
Thanks to the double meaning of si_fileid == 0 as "this is for a
delegation"--yes.
Post by Trond Myklebust
I'm not sure about whether or not overflows of current_ownerid and
current_delegid are as worrisome, but there appears to be no guarantee
against collisions...
Yeah.
Maybe we should make the opaque part out of a boot time and a single
64-bit counter. ?I don't see why we need the separate fileid and
stateownerid.
--b.
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
Steven Brudenell
2010-04-12 20:00:28 UTC
Permalink
Guys,

I'm sad that this never got any attention. My lab had to downgrade our
central fileserver away from nfs4 in order to solve this problem. :(

I do hope it gets solved in the future.

~Steve

On Fri, Feb 26, 2010 at 7:18 PM, Steven Brudenell
Post by Steven Brudenell
Nothing at all.
I thought we had determined the proximal cause, since you verified
that the stateid was created 20 hours earlier than the clientid.
I still can't explain why our nfs server keeps going back in time.
However, a sysadmin playing with the clock would cause this.
I'll be happy to debug this further, but I'll need some guidance. Is
there some kind of hook I can set to log whenever the system time
changes? Is there anything else I can do to debug the nfs code?
~Steve
Post by Steven Brudenell
Ping again (because my client machine crashed again). Did any change
ever get committed for this?
Unfortunately I think we still haven't determined a cause. ?Have you
found out anything more about the crashes?
--b.
Post by Steven Brudenell
~Steve
On Mon, Feb 1, 2010 at 3:04 PM, Steven Brudenell
Post by Steven Brudenell
Hey folks,
Just wondering if there has been any progress on this issue as yet. ?I
don't think I've gone two days without rebooting my client machine in
a while. :(
~Steve
Post by J. Bruce Fields
Post by Trond Myklebust
The handling of current_fileid appears to be buggy. There is no
protection against an overflow situation, so current_fileid == 0 is not
excluded. That would definitely give rise to "interesting" consequences.
Thanks to the double meaning of si_fileid == 0 as "this is for a
delegation"--yes.
Post by Trond Myklebust
I'm not sure about whether or not overflows of current_ownerid and
current_delegid are as worrisome, but there appears to be no guarantee
against collisions...
Yeah.
Maybe we should make the opaque part out of a boot time and a single
64-bit counter. ?I don't see why we need the separate fileid and
stateownerid.
--b.
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
Continue reading on narkive:
Loading...