[Fwd: Re: Permission problem with NFSv4 mount]

[Mike Brodbelt]

One way to check that would be to create a world-writeable directory on
the export, then create a file in it from the client as user "postgres"
on the client, then go back to the server and ls -l it to see what user
it was created as.

OK, so I changed permissions on the export and the mount point to
rwxrwxrwx, and remounted the NFS filesystem:-

# umount /db/data/
umount: /db/data/: not mounted
# chmod 777 /db/data/
# mount /db/data/
# ls -ld /db/data/
drwxrwxrwx 10 postgres postgres 25 2009-02-24 17:56 /db/data/

Even with world writable permissions though, I still get the error:-

# su - postgres
$ touch /db/data/testfile
touch: cannot touch `/db/data/testfile': Permission denied

Might also be interesting to watch the network with wireshark while you
do the new-file creation; of particular interest would be the rpc header
(especially the cred field) on the request that creates the file.

I've done a packet capture of the operations, and the tshark output is:-

1 0.000000 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
PUTFH;ACCESS;GETATTR
2 0.000689 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
1) <EMPTY> PUTFH;ACCESS;GETATTR
3 0.000699 10.0.0.39 -> 10.0.0.43 TCP 858 > nfs [ACK] Seq=157
Ack=241 Win=126 Len=0 TSV=96150336 TSER=176053765
4 0.000789 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
SETCLIENTID
5 0.001188 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
4) <EMPTY> SETCLIENTID
6 0.001221 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
SETCLIENTID_CONFIRM;PUTROOTFH;GETATTR
7 0.028545 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
6) <EMPTY> SETCLIENTID_CONFIRM;PUTROOTFH;GETATTR
8 0.028589 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
PUTFH;SAVEFH;OPEN;DELEGRETURN;Unknown
9 0.028695 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [SYN] Seq=0
Win=32804 Len=0 MSS=1460 TSV=176053768 TSER=0 WS=5
10 0.028702 10.0.0.39 -> 10.0.0.43 TCP 57740 > 62582 [SYN, ACK]
Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=96150343 TSER=176053768 WS=9
11 0.028794 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In 8)
<EMPTY> PUTFH;SAVEFH;OPEN
12 0.028803 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [ACK] Seq=1
Ack=1 Win=1049792 Len=0 TSV=176053768 TSER=96150343
13 0.028865 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
PUTFH;ACCESS;GETATTR
14 0.028994 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
13) <EMPTY> PUTFH;ACCESS;GETATTR[Malformed Packet]
15 0.032439 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [PSH, ACK]
Seq=1 Ack=1 Win=1049792 Len=72 TSV=176053769 TSER=96150343
16 0.032448 10.0.0.39 -> 10.0.0.43 TCP 57740 > 62582 [ACK] Seq=1
Ack=73 Win=6144 Len=0 TSV=96150344 TSER=176053769
17 0.032463 10.0.0.39 -> 10.0.0.43 TCP 57740 > 62582 [PSH, ACK]
Seq=1 Ack=73 Win=6144 [TCP CHECKSUM INCORRECT] Len=28 TSV=96150344
TSER=176053769
18 0.032538 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [ACK] Seq=73
Ack=29 Win=1049792 Len=0 TSV=176053769 TSER=96150344
19 0.065949 10.0.0.39 -> 10.0.0.43 TCP 858 > nfs [ACK] Seq=861
Ack=685 Win=126 Len=0 TSV=96150353 TSER=176053768

The packet for the file creation attempt is packet 11, and the details
of the RPC are:-

Credentials

Flavour: AUTH_UNIX
Machine name: <client hostname>
UID: 106
GID: 500
Auxiliary GIDs - GID: 500

Those UID and GID combinations correspond to the postgres user on the
client:-

$ id
uid=106(postgres) gid=500(postgres) groups=500(postgres)

The response in frame 11 has status of NFS4ERR_ACCES.

I'm still mystified as to what's going on. I can't provide many more
details on the server - it's a Sun Storage 7410 system, which is a
storage appliance that runs Solaris under the hood. The system provides
a web interface or a CLI for configuration, but there is no way for me
to get a shell on it, so I can't create files on it locally.

Mike

[J. Bruce Fields]

Post by Mike Brodbelt

One way to check that would be to create a world-writeable directory on
the export, then create a file in it from the client as user "postgres"
on the client, then go back to the server and ls -l it to see what user
it was created as.

OK, so I changed permissions on the export and the mount point to
rwxrwxrwx, and remounted the NFS filesystem:-
# umount /db/data/
umount: /db/data/: not mounted
# chmod 777 /db/data/
# mount /db/data/
# ls -ld /db/data/
drwxrwxrwx 10 postgres postgres 25 2009-02-24 17:56 /db/data/
Even with world writable permissions though, I still get the error:-
# su - postgres
$ touch /db/data/testfile
touch: cannot touch `/db/data/testfile': Permission denied

And from the below it looks like the client is indeed using auth_unix
with the correct uid and gid. So, I'm stumped. Sorry. It looks like a
server-side problem to me.

--b.

Post by Mike Brodbelt

Might also be interesting to watch the network with wireshark while you
do the new-file creation; of particular interest would be the rpc header
(especially the cred field) on the request that creates the file.

I've done a packet capture of the operations, and the tshark output is:-
1 0.000000 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
PUTFH;ACCESS;GETATTR
2 0.000689 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
1) <EMPTY> PUTFH;ACCESS;GETATTR
3 0.000699 10.0.0.39 -> 10.0.0.43 TCP 858 > nfs [ACK] Seq=157
Ack=241 Win=126 Len=0 TSV=96150336 TSER=176053765
4 0.000789 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
SETCLIENTID
5 0.001188 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
4) <EMPTY> SETCLIENTID
6 0.001221 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
SETCLIENTID_CONFIRM;PUTROOTFH;GETATTR
7 0.028545 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
6) <EMPTY> SETCLIENTID_CONFIRM;PUTROOTFH;GETATTR
8 0.028589 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
PUTFH;SAVEFH;OPEN;DELEGRETURN;Unknown
9 0.028695 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [SYN] Seq=0
Win=32804 Len=0 MSS=1460 TSV=176053768 TSER=0 WS=5
10 0.028702 10.0.0.39 -> 10.0.0.43 TCP 57740 > 62582 [SYN, ACK]
Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=96150343 TSER=176053768 WS=9
11 0.028794 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In 8)
<EMPTY> PUTFH;SAVEFH;OPEN
12 0.028803 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [ACK] Seq=1
Ack=1 Win=1049792 Len=0 TSV=176053768 TSER=96150343
13 0.028865 10.0.0.39 -> 10.0.0.43 NFS V4 COMPOUND Call <EMPTY>
PUTFH;ACCESS;GETATTR
14 0.028994 10.0.0.43 -> 10.0.0.39 NFS V4 COMPOUND Reply (Call In
13) <EMPTY> PUTFH;ACCESS;GETATTR[Malformed Packet]
15 0.032439 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [PSH, ACK]
Seq=1 Ack=1 Win=1049792 Len=72 TSV=176053769 TSER=96150343
16 0.032448 10.0.0.39 -> 10.0.0.43 TCP 57740 > 62582 [ACK] Seq=1
Ack=73 Win=6144 Len=0 TSV=96150344 TSER=176053769
17 0.032463 10.0.0.39 -> 10.0.0.43 TCP 57740 > 62582 [PSH, ACK]
Seq=1 Ack=73 Win=6144 [TCP CHECKSUM INCORRECT] Len=28 TSV=96150344
TSER=176053769
18 0.032538 10.0.0.43 -> 10.0.0.39 TCP 62582 > 57740 [ACK] Seq=73
Ack=29 Win=1049792 Len=0 TSV=176053769 TSER=96150344
19 0.065949 10.0.0.39 -> 10.0.0.43 TCP 858 > nfs [ACK] Seq=861
Ack=685 Win=126 Len=0 TSV=96150353 TSER=176053768
The packet for the file creation attempt is packet 11, and the details
of the RPC are:-
Credentials
Flavour: AUTH_UNIX
Machine name: <client hostname>
UID: 106
GID: 500
Auxiliary GIDs - GID: 500
Those UID and GID combinations correspond to the postgres user on the
client:-
$ id
uid=106(postgres) gid=500(postgres) groups=500(postgres)
The response in frame 11 has status of NFS4ERR_ACCES.
I'm still mystified as to what's going on. I can't provide many more
details on the server - it's a Sun Storage 7410 system, which is a
storage appliance that runs Solaris under the hood. The system provides
a web interface or a CLI for configuration, but there is no way for me
to get a shell on it, so I can't create files on it locally.
Mike
--
Director of Technical Operations and Infrastructure
Last.fm Ltd. | http://www.last.fm/ | Email: mike at last.fm
Office: [phone] | Mobile: [phone]
_______________________________________________
NFSv4 mailing list
NFSv4 at linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

[Tom Haynes]

Post by J. Bruce Fields
And from the below it looks like the client is indeed using auth_unix
with the correct uid and gid. So, I'm stumped. Sorry. It looks like a
server-side problem to me.
--b.

Which is why I'd like to know what the share options are on the server.

Post by J. Bruce Fields

I have configured the idmap domain on the storage appliance, and in
/etc/idmapd.conf on the client, and have started the idmapd service on
the storage appliance, and on the Linux client.
The export mounts fine on the client, however I cannot write to it as a
non root user. As root, writes work fine (root squash on the server has
been disabled for this export). So:-

What is the export?

I'm not sure what you mean... It's exported r/w, with anon users mapped
to nobody, and root access explicitly allowed from the client in question.

'root squash' is a Linux term, but the second reply sounds better.

What I would like to see is the full share from the server.

Things to look for:

1) Is the root access by hostname or by ip?
2) If hostname, is it fully qualified?
3) What is the order for hosts in resolv.conf?
4) If it is by IP, did you use the correct notation?

Mike, you can also try bringing this up on nfs-discuss at opensolaris.org.
I'll ask the same questions there, but just via my sun.com account. :->

[Mike Brodbelt]

Post by Tom Haynes

Post by J. Bruce Fields
And from the below it looks like the client is indeed using auth_unix
with the correct uid and gid. So, I'm stumped. Sorry. It looks like a
server-side problem to me.
--b.

Which is why I'd like to know what the share options are on the server.

Post by J. Bruce Fields

I have configured the idmap domain on the storage appliance, and in
/etc/idmapd.conf on the client, and have started the idmapd service on
the storage appliance, and on the Linux client.
The export mounts fine on the client, however I cannot write to it as a
non root user. As root, writes work fine (root squash on the server has
been disabled for this export). So:-

What is the export?

I'm not sure what you mean... It's exported r/w, with anon users mapped
to nobody, and root access explicitly allowed from the client in question.

'root squash' is a Linux term, but the second reply sounds better.
What I would like to see is the full share from the server.
show

Properties:
aclinherit = restricted
aclmode = discard
atime = false (inherited)
casesensitivity = mixed
checksum = fletcher2 (inherited)
compression = off (inherited)
compressratio = 100
copies = 1 (inherited)
creation = Fri Feb 06 2009 17:57:38 GMT+0000 (UTC)
mountpoint = /export/globalslave (inherited)
normalization = none
quota = 0
quota_snap = true
readonly = false (inherited)
recordsize = 8K (inherited)
reservation = 0
reservation_snap = true
secondarycache = all (inherited)
nbmand = false (inherited)
sharesmb = off (inherited)
sharenfs = rw,root=<client hostname> (inherited)
snapdir = hidden (inherited)
utf8only = true
vscan = false (inherited)
sharedav = off (inherited)
shareftp = off (inherited)
space_data = 1.03T
space_unused_res = 0
space_snapshots = 0
space_available = 9.30T
space_total = 1.03T
root_group = postgres
root_permissions = 777
root_user = postgres

Post by Tom Haynes
1) Is the root access by hostname or by ip?

The access is setup by hostname.

Post by Tom Haynes
2) If hostname, is it fully qualified?

Yes.

Post by Tom Haynes
3) What is the order for hosts in resolv.conf?

That's not accessible to me on the appliance. I can ping the client from
the appliance command line though, which demonstrates that DNS
resolution is working.

Post by Tom Haynes
4) If it is by IP, did you use the correct notation?
Mike, you can also try bringing this up on nfs-discuss at opensolaris.org.
I'll ask the same questions there, but just via my sun.com account. :->

I may bounce it there next. I've done a fair bit of poring over the RFC
and network traces, and I'm not getting anywhere at the moment.

Mike

[Tom Haynes]

Post by Tom Haynes
2) If hostname, is it fully qualified?

Yes.

The typical things I would be looking for here is whether there is a
match between the hostname used in the access list and that
of the client. I'd also want to make sure that idmapping is working
correctly on the server.

Things to try:

1) Use anon=0 instead of root=hostname.

If this works, it shows that the problem is with the hostname. If it
fails to work, it probably shows that the problem is in idmapping.

2) Use root=@hostip/32

I'm not sure what made it to the OS you are running, but this notation
will work.

If this works, it shows that the problem is with the hostname.

3) Try rw=hostname and see if regular users can write or not. Check the
resulting owners.

I'll look to see if I can get my hands on some of the appliance docs to
see if there are other ways to troubleshoot this problem.

The above steps are what I would do in OpenSolaris, albeit with either a
zfs or share command.

[Tom Haynes]

Mike,

I pulled down the Sun Unified Storage Simulator -- see
http://www.sun.com/storage/disk_systems/unified_storage/resources.jsp
and got a system up and running. Unfortunately I haven't been having as
good a luck getting Lenny to work for me in a Virtual Machine.

I've tried to connect from my Fedora 8 client -- 2.6.26.5-28.fc8 -- and
I'm not having the issues you report.

Okay, I've got a Lenny system going.

On the server, if I change 'Anonymous user mapping' from 'nobody' to
'braves', then I' m not able to write, even as root.

But I expect that as my share is:

sharenfs = rw,root=adept.internal.excfb.com

and I'm coming in from pseries.internal.excfb.com

Actually, with the change of anon, I have:

sharenfs =
rw,anon=braves,root=adept.internal.excfb.com

With this share:
sharenfs =
rw,anon=braves,root=pseries.internal.excfb.com:adept.internal.excfb.com

I get:

tdh at pseries:/nfsv4$ sudo touch jjj
touch: cannot touch `jjj': Permission denied
tdh at pseries:/nfsv4$ touch jjj
touch: cannot touch `jjj': Permission denied
tdh at pseries:/nfsv4$ uname -a
Linux pseries 2.6.26-1-686 #1 SMP Sat Jan 10 18:29:31 UTC 2009 i686
GNU/Linux

where pseries is my Lenny system.

On my Fedora client, I get:

[tdh at adept /nfsv4]> touch adept
[tdh at adept /nfsv4]> ls -la adept
-rw-r--r-- 1 tdh nobody 0 2009-03-03 20:09 adept
[tdh at adept /nfsv4]> sudo touch root_adept
[tdh at adept /nfsv4]> ls -la root_adept
-rw-r--r-- 1 braves nobody 0 2009-03-03 20:10 root_adept
[tdh at adept /nfsv4]> uname -a
Linux adept.internal.excfb.com 2.6.26.5-28.fc8 #1 SMP Sat Sep 20
09:32:58 EDT 2008 i686 i686 i386 GNU/Linux

So, what is different between the two clients? Well, I made adept a NIS
server for this testing and pseries is not a NIS client.

Let me fix that.

And that made things start to work for me.... (Note, even without NIS,
the Fedora install worked for me.)

tdh at pseries:/nfsv4$ touch fook
tdh at pseries:/nfsv4$ ls -la fook
-rw-r--r-- 1 4294967294 4294967294 0 2009-03-03 20:14 fook

My idmap is off for this machine as well, which I don't understand why.

[Mike Brodbelt]

Post by Tom Haynes
tdh at pseries:/nfsv4$ sudo touch jjj
touch: cannot touch `jjj': Permission denied
tdh at pseries:/nfsv4$ touch jjj
touch: cannot touch `jjj': Permission denied
tdh at pseries:/nfsv4$ uname -a
Linux pseries 2.6.26-1-686 #1 SMP Sat Jan 10 18:29:31 UTC 2009 i686
GNU/Linux
where pseries is my Lenny system.
[tdh at adept /nfsv4]> touch adept
[tdh at adept /nfsv4]> ls -la adept
-rw-r--r-- 1 tdh nobody 0 2009-03-03 20:09 adept
[tdh at adept /nfsv4]> sudo touch root_adept
[tdh at adept /nfsv4]> ls -la root_adept
-rw-r--r-- 1 braves nobody 0 2009-03-03 20:10 root_adept
[tdh at adept /nfsv4]> uname -a
Linux adept.internal.excfb.com 2.6.26.5-28.fc8 #1 SMP Sat Sep 20
09:32:58 EDT 2008 i686 i686 i386 GNU/Linux
So, what is different between the two clients? Well, I made adept a NIS
server for this testing and pseries is not a NIS client.
Let me fix that.

That's very interesting - thanks a lot for doing that testing for me.

However, I'm still confused about what I need to do to get it to work
for me - we're not using NIS anywhere in the network, and the client
machine has no NIS packages installed (except the libs that come as part
of the glibc package).

Post by Tom Haynes
And that made things start to work for me.... (Note, even without NIS,
the Fedora install worked for me.)

That's weird - I wonder what's different with Lenny's NFSv4 support? I
don't have a Fedora install to hand, but trying it with CentOS produces
similar results to what you're seeing - I can write files as the
postgres user, but they appear as owned by nobody:-

$ ls -l centos_file
-rw-r--r-- 1 nobody nobody 0 Mar 6 01:30 centos_file
$ id
uid=106(postgres) gid=500(postgres) groups=500(postgres)

So it looks like idmapping is broken for both, but Debian has a serious
bug somewhere... Which is rather a pain, as all my potential NFS clients
are Debian boxes.

Post by Tom Haynes
tdh at pseries:/nfsv4$ touch fook
tdh at pseries:/nfsv4$ ls -la fook
-rw-r--r-- 1 4294967294 4294967294 0 2009-03-03 20:14 fook
My idmap is off for this machine as well, which I don't understand why.

So, like me, you're being mapped to nfsnobody, even though the client
is running idmapd?

Mike

[Tom Haynes]

Post by Mike Brodbelt

Post by Tom Haynes
tdh at pseries:/nfsv4$ touch fook
tdh at pseries:/nfsv4$ ls -la fook
-rw-r--r-- 1 4294967294 4294967294 0 2009-03-03 20:14 fook
My idmap is off for this machine as well, which I don't understand why.

So, like me, you're being mapped to nfsnobody, even though the client
is running idmapd?
Mike

Yes.