![首頁 Logo Keith ]p !!](/themes/mytheme/logo.png){kind=logo}
最後更新: 2023-09-05
目錄
-
安裝
Debian 5
Centos 6
Centos 7 - NFS Version
-
Server 設定 Share Folder
- /etc/exportfs
- exportfs
介紹
詳見:
NFS 共分為四個版本
- NFS V4.1
- NFS V4 rfc3010
- NFS V3 rfc1813
- NFS V2 rfc1094
當然, 最新版功能比較強啦
NFS Server 與 Client 之間是通過 RPC 來實現通訊的, 並透過不同的程式實現不同的工作
Daemon
- portmap: # port mapping (RPC)
- rpc.nfsd: # 管理登入
- rpc.mountd: # 負責NFS的檔案系統
- rpc.lockd: (optional) # 它會跟 "service nfs start" 一起 start
- rpc.statd: (optional) # /etc/init.d/nfslock
安裝
Debian 5
nfs-kernel-server - support for NFS kernel server
apt-get install nfs-kernel-server
Centos 6
yum -y install nfs-utils
提供的program:
/usr/sbin/rpc.svcgssd
/usr/sbin/rpc.nfsd
/usr/sbin/exportfs
/usr/sbin/rpc.mountd
Setting
/etc/sysconfig/nfs
# rpc.mountd
# Define which protocol versions mountd
MOUNTD_NFS_V1="no"
MOUNTD_NFS_V2="no"
MOUNTD_NFS_V3="yes"
# rpc.nfsd
# Turn off v2 and v3 protocol support
RPCNFSDARGS="-N 2 -N 3"
# total number of NFS proccess (number of clients that access same time)
RPCNFSDCOUNT=8
# 指定不同 Service 用什麼 port
LOCKD_TCPPORT=32803
MOUNTD_PORT=892
STATD_PORT=662
chkconfig nfs on
service nfs start
service nfs status
rpc.svcgssd is stopped rpc.mountd (pid 4649) is running... nfsd (pid 4716 4715 4714 4713 4712 4711 4710 4709) is running... rpc.rquotad (pid 4645) is running...
rpc.mountd (892)
The rpc.mountd daemon implements the server side of the NFS MOUNT protocol
rpc.mountd uses these ACL (/etc/exports) to determine whether an NFS client is permitted to access a given file system.
It provides an ancillary service needed to satisfy mount requests by NFS clients.
rpc.mountd returns an NFS file handle for the export's root directory to the client.
rpc.mountd daemon registers every successful MNT request by adding an entry to the "/var/lib/nfs/rmtab"
NFS MOUNT protocol
MNT (mount an export)
- pathname of the root directory of the export
- sender's IP address
UMNT (unmount an export)
rpc.nfsd ( TCP / UDP port: 2049)
implements the user level part of the NFS service
The user space program merely specifies
* sort of sockets the kernel service should listen on,
* what NFS versions it should support,
* how many kernel threads it should use.
The main functionality is handled by the nfsd kernel module
lsmod | grep nfsd
Centos 7
yum -y install nfs-utils
# nfs 依賴 rpcbind
當 nfs 唔 work 時, 可以用以下 cmd 去 check
systemctl status rpcbind
# re-generates config file
# configure file: /run/sysconfig/nfs-utils
systemctl restart nfs-config
cat /run/sysconfig/nfs-utils
RPCNFSDARGS=" 8" RPCMOUNTDARGS="" STATDARGS="" SMNOTIFYARGS="" RPCIDMAPDARGS="" GSSDARGS="" BLKMAPDARGS="" GSS_USE_PROXY="yes"
# status
systemctl status nfs
NFS Version
NFSv3
The NFS version 2 and 3 protocols are stateless.(The NFSv4 protocol introduces state)
NFSv4
NFS Version 4 combines the disparate NFS protocols (stat, NLM, mount, ACL, and NFS)
into a single protocol specification to allow better compatibility with network firewalls.
* mountd, statd, and lockd are not required in a pure NFSv4 environment.
* single target port (2049/tcp)
* NFSv4 clients are required to renew leases on files and filesystems on regular basis.
=> This activity keeps the TCP session active.
Stateful
Opening, locking, reading, and writing, carry state information that notify the server of the intentions on the object by the client.
The server can then return information to a client about other clients that also have intentions on the same object.
Locking
- Support for byte-range locking
- Locking in NFSv4 is lease-based
String data
All "string data" used by the protocol is represented in UTF-8 as it crosses the network.
User and group information is passed between the client and server in string form, not as numeric values as in previous versions.
Remark
NFS version 4 performance might be slower than with NFS version 3 for many applications.
The performance impact varies significantly depending on which new functions you use.
NFSv4.1
pNFS extension
Server 設定 Share Folder
/etc/exportfs
[共享的目錄] [Client1_IP(設定1,設定2)] Client2_IP *(ro)
當設定漏空時, 就會用 default 的 sync,ro,root_squash,no_delay
IP
address/netmask
netmask: /255.255.252.0' or `/22'
Opts
ro / rw 只讀 (Default) / 讀寫
sync / async sync (Default)
async: reply to client requests as soon as it has processed the request and handed it off to the local file system
secure / insecure 通過 port 1024以下 TCP/IP Port 溝通 (Default) / 1024 以上
wdelay（Default） Causes the NFS server to delay writing to the disk if it suspects another write request is imminent.
no_wdelay 如果多個用戶要寫入NFS目錄，則立即寫入，當使用async時，無需此設置。
hide 在NFS共享目錄中, 不個別共享其子目錄 (Default)
nohide 共享NFS目錄的子目錄
subtree_check 如果共享/usr/bin之類的子目錄時，強制NFS檢查父目錄的權限（Default）
no_subtree_check 與 subtree_check 相對, 不檢查父目錄權限
all_squash # 共享文件的UID和GID映射到 nobody 和 nogroup 上
no_all_squash # 保留共享文件的UID和GID（Default）
root_squash # root用戶的所有請求映射成如anonymous用戶一樣的權限(Default）
no_root_squash # root用戶具有根目錄的完全管理訪問權限
anonuid=xxx # anonymous用戶 UID (nobody 65534)
anongid=xxx # anonymous用戶 GID (nogroup 65534)
pnfs # pNFS extension. NFSv4.1 or higher and the fs supports pNFS
# pNFS clients can bypass the server and perform I/O directly to storage devices
Help
man exports
exportfs
可以在不影響 /etc/exports 去建立或取消分施
可用選項:
-a All / Export
-u Unexport
-r Reexport
-v verbose
initialized share
exportfs -a
讀取 /etc/exports 去生成 /var/lib/nfs/etab 文件
mountd 會去讀它, 去決定 Client 的 Access
etab 包括原整設定
/home/share 192.168.123.112(rw,sync,wdelay,hide,nocrossmnt,secure,
root_squash,no_all_squash,no_subtree_check,secure_locks,acl,
anonuid=65534,anongid=65534)
支援 address/netmask
查看什麼 export 了
exportfs
/home/share 192.168.123.112
看 export opt
exportfs -v
/home/vhosts/X/api/public_html/uploads 172.16.11.11(rw,sync,wdelay,hide,no_subtree_check,sec=sys,secure,no_root_squash,no_all_squash)
重新讀取 /etc/exports 文件
exportfs -r
取消分享
取消所有分享
exportfs -au
單是 IP:/usr/tmp 被取消
exportfs -u IP:/usr/tmp
查看 Network 及 Statistics
lsof -i|grep rpc
nfsstat [-l]
-l = list form # Print information in list form
-c, --client # Print only client-side statistics.
-s, --server # Print only server-side statistics.
Server rpc stats: calls badcalls badclnt badauth xdrcall 0 0 0 0 0
-l
nfs v3 client total: 604 ------------- ------------- -------- nfs v3 client getattr: 66 nfs v3 client setattr: 4 nfs v3 client lookup: 16 nfs v3 client access: 11 nfs v3 client write: 481 nfs v3 client create: 8 nfs v3 client remove: 3 nfs v3 client readdirplus: 4 nfs v3 client fsinfo: 6 nfs v3 client pathconf: 1 nfs v3 client commit: 4
* 每次操作都是有 commit 的
在 /proc 有關的 file
/proc/fs/nfsd/ 內有
export_features exports filehandle max_block_size nfsv4gracetime nfsv4leasetime nfsv4recoverydir pool_stats pool_threads portlist reply_cache_stats supported_krb5_enctypes threads unlock_filesystem unlock_ip versions
/proc/fs/nfs/exports
# Version 1.2 # Path Client(Flags) # IPs
Remote Procedure Call
告訴 Client 每個 NFS 功能所對應的 port number
(因為 NFS 用來傳輸的 port 是隨機小於 1024 的)
NFS 隨機取用Port ---注冊--> RPC
Notes
Portmapper is a service that is utilized for mapping network service ports
to RPC (Remote Procedure Call) program numbers.
Default port: 111/TCP/UDP
rpcinfo 查看什麼 program 注冊了 rpc
- -p Probe
Example:
rpcinfo -p localhost
Default:
[root@hyps2 root]# rpcinfo -p localhost
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 826 status
100024 1 tcp 829 status啟用 nfs 後:
program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 826 status 100024 1 tcp 829 status 100011 1 udp 895 rquotad 100011 2 udp 895 rquotad 100011 1 tcp 898 rquotad 100011 2 tcp 898 rquotad 100003 2 udp 2049 nfs 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100003 2 tcp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100021 1 udp 57975 nlockmgr 100021 3 udp 57975 nlockmgr 100021 4 udp 57975 nlockmgr 100021 1 tcp 32779 nlockmgr 100021 3 tcp 32779 nlockmgr 100021 4 tcp 32779 nlockmgr 100005 1 udp 911 mountd 100005 1 tcp 914 mountd 100005 2 udp 911 mountd 100005 2 tcp 914 mountd 100005 3 udp 911 mountd 100005 3 tcp 914 mountd
再開 Service
/etc/init.d/portmap restart
Default:
portmap 0:off 1:off 2:off 3:off 4:off 5:off 6:off
P.S.
RPC 若重新啟動時, 原本註冊的資料會不見
與 rpcgssd, rpcidmapd, rpcsvcgssd 無關
rpcgssd
The rpcsec_gss protocol gives a means of using the gss-api generic security api to provide security for protocols using rpc (in particular, nfs). Before exchanging any rpc requests using rpcsec_gss, the rpc client must first establish a security context. The linux kernel's implementation of rpcsec_gss depends on the userspace daemon rpc.gssd to establish security contexts. The rpc.gssd daemon uses files in the rpc_pipefs filesystem to communicate with the kernel.
-f Runs rpc.gssd in the foreground and sends output to stderr (as opposed to syslogd)
rpcidmapd
rpc.idmapd is the NFSv4 ID <-> name mapping daemon. It provides functionality to the NFSv4 kernel client and server, to which it communicates via upcalls, by translating user and group IDs to names, and vice versa.
rpcsvcgssd
The rpcsec_gss protocol gives a means of using the gss-api generic security api to provide security for protocols using rpc (in particular, nfs). Before exchanging any rpc requests using rpcsec_gss, the rpc client must first establish a security context with the rpc server. The linux kernel's implementation of rpcsec_gss depends on the userspace daemon rpc.svcgssd to handle context establishment on the rpc server. The daemon uses files in the proc filesystem to communicate with the kernel.
用戶身份
Server 上是用 Client 的提供的 UID 及 GID 來認證身份
苦在 Server 上並沒有 Client 所提供的 UID 時, 系統就會用 nobody 去取代
所以, 用 NIS 去保持身份一致是個不錯的選擇
Client Side
安裝:
apt-get install nfs-common
提供工具:
- /sbin/showmount
- /sbin/mount.nfs
- /sbin/umount.nfs
- /usr/sbin/nfsstat
查看資源:
showmount [-e[Server_IP]]
- -a client IP and mounted directory
- -e Export list
- no options 什麼 Client 連上來
i.e. localhost
showmount
Hosts on debianA: 192.168.123.112
i.e. ip: 192.168.123.111
showmount -e 192.168.123.111
Export list for 192.168.123.111: /home/share (everyone)
掛載:
mount Server_IP:/home/share /mnt/share
可用設定:
intr # 允許通知中斷一個NFS調用
udp/tcp
TCP advantage:
On lossy networks, a single dropped packet can be retransmitted, without the retransmission of the entire RPC request
noac # 關閉cache機制
retry=n
hard / soft # Default: hard
*hard 不斷的嘗試
*soft 收到錯誤信息後終止mount嘗試(Default)
An NFS request after retrans retransmissions have been sent, causing the NFS client to return an error to the calling application.
Example:
mount -t nfs -o soft,udp,noatime Server_IP:/DIR /Local_DIR
開機時自動掛載
在 /etc/fstab 加下
Server_IP:/home/share /mnt/share nfs rsize=8192,wsize=8192,timeo=14
防火牆 (Running NFS Behind a Firewall)
Allow ACL
# Allow TCP and UDP port 2049 for NFS. # Allow TCP and UDP port 111 (rpcbind(portmap)) -A INPUT -p tcp -m state --state NEW -m multiport --dport 111,2049 -j ACCEPT -A INPUT -p udp -m state --state NEW -m multiport --dport 111,2049 -j ACCEPT
* NFS requires rpcbind, which dynamically assigns ports for RPC services
* NFS4 開 2 個 port 已經夠 (111,2049)
Port:
1-1024: mountd
禁止某 IP 登入
我們可以透過hosts.deny 文件 或 iptables 去實施
在 hosts.deny上實施:
在 /etc/hosts.deny 內加入禁止的 host
portmap:IP
在iptables 上實施:
iptables -A INPUT -i eth0 -p TCP -s IP --dport 111 -j DROP
Dynamically port to Static port
Centos7
# control which ports the required RPC services run on
/etc/sysconfig/nfs
# mountd (rpc.mountd) [TCP/UDP] MOUNTD_PORT=892 # status (rpc.statd) [TCP/UDP] STATD_PORT=662
# Restart service
systemctl restart nfs
systemctl restart rpc-statd
lockd
# 方法 1
/etc/modprobe.d/lockd.conf
# 方法 2
vim /etc/sysctl.conf
fs.nfs.nlm_tcpport=57269 fs.nfs.nlm_udpport=57269
# 套用 setting
sysctl -p
checking
cat /proc/sys/fs/nfs/nlm_*port
Checking
# report RPC information
rpcinfo -p
OR
rpcinfo -p | awk '{print $3 "\t" $4 "\t" $5}' | sort -u
# by log
tail -f /var/log/messages
firewalld
rich-rule
firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="tcp" port="111" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="udp" port="111" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="tcp" port="2049" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="udp" port="2049" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="tcp" port="892" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="udp" port="892" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="tcp" port="662" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="udp" port="662" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="tcp" port="57269" accept' firewall-cmd --permanent \ --add-rich-rule='rule family="ipv4" source address="172.16.11.11" port protocol="udp" port="57269" accept' firewall-cmd --reload
service.xml
firewall-cmd --permanent --add-service=nfs firewall-cmd --permanent --add-service=rpcbind firewall-cmd --permanent --add-service=mountd firewall-cmd --reload
nfs
<?xml version="1.0" encoding="utf-8"?> <service> <short>NFS4</short> <description>...</description> <port protocol="tcp" port="2049"/> </service>
mountd
<?xml version="1.0" encoding="utf-8"?> <service> <short>mountd</short> <description>NFS Mount Lock Daemon</description> <port protocol="tcp" port="20048"/> <port protocol="udp" port="20048"/> </service>
NFS Version
# Default: NFS uses the highest supported version by the kernel and mount command.
Checking
nfsstat -m
... vers=4 ...
設定
vers=3|4
Performance
Client
vers
vers=3
retrans / timeo
retrans=3,timeo=7
the client will resend the RPC packet 3 times and each time it will wait for 7/10 seconds
# statistics for retranmission of packets
nfsstat
Client rpc stats: calls retrans authrefrsh 133 2 133 Client nfs v4: null read write commit open open_conf 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% ...
wsize / rsize
wsize 和 rsize 設定了 SERVER 與 CLIENT 之間往來數據塊 (chunks of data) 的大小, 單位為 bytes
NFS V2 最大值 8K (8192 byte)
NFS V3 最大值 64K (65536 byte)
* 不同的操作系統有不同的最佳數值
Example:
mount -t nfs -o rsize=1024,wsize=1024,noatime,nodiratime Server_IP:/DIR /Local_DIR
Server
number of nfs threads
When a request is made for an NFS service, the Services for UNIX NFS server generates a thread to handle the request.
Each thread can process one NFS request. A large pool of threads can allow a server to handle more NFS requests in parallel.
A dual-processor server typically uses 20 threads.
Centos Setting
/etc/sysconfig/nfs
# the number of threads that will be started.
RPCNFSDCOUNT=8# Checking
grep th /proc/net/rpc/nfsd
# first: total number of NFS server threads started.
# second: indicates whether at any time all of the threads were running at once.
# remaining numbers: a thread count time histogram.
mount
# tcp with nfsv4
mount -t nfs -o proto=tcp,vers=4 192.168.88.210:/home/data /home/data
Statistics - /proc/net/rpc/nfsd
有關 Packages
nfs-common - NFS support files common to client and server
Adding new user `statd' (UID 108) with group `nogroup'
showmount
nfsboot - Allow clients to boot over the network
Linux kernel's "nfsroot=" boot option
nfsroot=192.168.2.72:/tftpboot/nfsroot,v3
/etc/dhcpd.conf settings for the client machine:
option root-path "/tftpboot/nfsroot,v3";
nfsbooted - Prepares your image for nfs boot
unfs3 - User-space NFSv3 Server
nfswatch
nfs4-acl-tools
Disable ubuntu nfs and portmap service
# service portmap stop
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1583/rpcbind
# service nfs-kernel-server stop
tcp 0 0 0.0.0.0:52662 0.0.0.0:* LISTEN 2891/rpc.mountd tcp 0 0 0.0.0.0:35834 0.0.0.0:* LISTEN 2891/rpc.mountd tcp 0 0 0.0.0.0:34522 0.0.0.0:* LISTEN 1695/rpc.statd tcp 0 0 0.0.0.0:55237 0.0.0.0:* LISTEN 2891/rpc.mountd
Troubleshoot
Error 1:
Starting NFS daemon: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused) rpc.nfsd: unable to set any sockets for nfsd
Fix 1:
service rpcbind restart
chkconfig rpcbind on
Error 2:
rsync: chown "PATH" failed: Operation not permitted (1)
Fix 2:
改用 NFS v3
Error 3:
exportfs: internal: no supported addresses in nfs_client exportfs: 192.168.1.252:/home/vm/migrate: No such file or directory
Fix 3:
/etc/init.d/nfs start
Remark
rpc.statd
A daemon that listens for reboot notifications from other hosts,
and manages the list of hosts to be notified when the local system reboots
After an NFS client reboots, an NFS server must release all file locks held by applications that were running on that client.
After a server reboots, a client must remind the server of file locks held by applications running on that client.
NSM state number for this host
# NSM (Network Status Monitor) protocol is used to notify NFS peers of reboots.
/var/lib/nfs/statd/state
directory containing monitor list
ls /var/lib/nfs/statd/sm
