oVirt - NFS - weird problem...

[Ve0]

Hello!
I have some problems with NFS. I have a oVirt self-hosted node. I have ovirtmgmt - oVirt brige interface. On hypervisor I install NFS server with fixed ports in config, firewall is disabled. And I need to connect nfs share to virtual machine...

Code: Select all

[root@vm ~]# showmount -e 192.168.0.99
Export list for vm.dazab.local:
/mnt/backup      192.168.0.0/24
/mnt/storage     192.168.0.0/24
/mnt/oVirt/vmimg 192.168.0.0/24
/mnt/oVirt/vmov  192.168.0.0/24
/mnt/oVirt/ssd   192.168.0.0/24
/mnt/oVirt/iso   192.168.0.0/24
[root@vm ~]#

This is from nfs server. On VM looks like:

Code: Select all

[root@web ~]# showmount -e 192.168.0.99
rpc mount export: RPC: Unable to receive; errno = No route to host
[root@web ~]#

Nmap from nfs server:

Code: Select all

[root@vm ~]# nmap 192.168.0.99

Starting Nmap 6.40 ( http://nmap.org ) at 2017-05-10 12:36 PDT
Nmap scan report for vm.dazab.local (192.168.0.99)
Host is up (0.0000070s latency).
Not shown: 991 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
111/tcp  open  rpcbind
2049/tcp open  nfs
5900/tcp open  vnc
5901/tcp open  vnc-1
5902/tcp open  vnc-2
5903/tcp open  vnc-3
5904/tcp open  unknown
5906/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.07 seconds
[root@vm ~]#

But when I scan from web server, I got just this one:

Code: Select all

[root@web ~]# nmap 192.168.0.99

Starting Nmap 7.40 ( https://nmap.org ) at 2017-05-10 15:37 EDT
Stats: 0:00:00 elapsed; 0 hosts completed (0 up), 1 undergoing ARP Ping Scan
ARP Ping Scan Timing: About 100.00% done; ETC: 15:37 (0:00:00 remaining)
Nmap scan report for vm.dazab.local (192.168.0.99)
Host is up (0.000077s latency).
Not shown: 915 filtered ports, 77 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
111/tcp  open  rpcbind
5900/tcp open  vnc
5901/tcp open  vnc-1
5902/tcp open  vnc-2
5903/tcp open  vnc-3
5904/tcp open  unknown
5906/tcp open  unknown
MAC Address: 00:1E:67:14:F3:19 (Intel Corporate)

Nmap done: 1 IP address (1 host up) scanned in 3.71 seconds
[root@web ~]#

Port #2049 is disappeared...

And another thing, I have another server in network on ubuntu server, so sometimes I can't see it from any VM based on oVirt.

Please, help! What I did wrong!?

[hunter86_bg]

Enable the mound firewall service and show the output of netstat and

Code: Select all

systemctl status nfs-server

[Ve0]

I forget to tell: I have 2 lans in bond 5 and oVirt bridge configured on bond. All oVirt VMs disks works on NFS. And NFS server works:

Code: Select all

[root@vm ~]# df -h
Filesystem                                       Size  Used Avail Use% Mounted on
/dev/sdd3                                         33G  3.2G   30G  10% /
devtmpfs                                          16G     0   16G   0% /dev
tmpfs                                             16G   24K   16G   1% /dev/shm
tmpfs                                             16G   50M   16G   1% /run
tmpfs                                             16G     0   16G   0% /sys/fs/cgroup
/dev/sdd1                                       1014M  216M  799M  22% /boot
/dev/mapper/2a904e08b00d00000p1                  550G  315G  207G  61% /mnt/oVirt
/dev/mapper/2ab67c28300d00000p1                  1.8T  466G  1.3T  27% /mnt/backup
/dev/mapper/3600508e00000000095dae96fb51d0e00p1  290G   31G  245G  12% /mnt/oVirt/ssd
/dev/mapper/2a845073a00d00000p1                  7.3T  2.2T  4.7T  32% /mnt/storage
192.168.0.99:/mnt/oVirt/vmov                     550G  315G  207G  61% /rhev/data-center/mnt/192.168.0.99:_mnt_oVirt_vmov
192.168.0.99:/mnt/oVirt/vmimg                    550G  315G  207G  61% /rhev/data-center/mnt/192.168.0.99:_mnt_oVirt_vmimg
192.168.0.99:/mnt/oVirt/iso                      550G  315G  207G  61% /rhev/data-center/mnt/192.168.0.99:_mnt_oVirt_iso
192.168.0.99:/mnt/oVirt/ssd                      290G   31G  245G  12% /rhev/data-center/mnt/192.168.0.99:_mnt_oVirt_ssd
tmpfs                                            3.1G     0  3.1G   0% /run/user/0
[root@vm ~]# 

But just in case:

Code: Select all

[root@vm ~]# systemctl status nfs-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2017-04-28 11:33:02 PDT; 1 weeks 5 days ago
 Main PID: 1333 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Apr 28 11:33:01 vm.dazab.local systemd[1]: Starting NFS server and services...
Apr 28 11:33:02 vm.dazab.local systemd[1]: Started NFS server and services.
[root@vm ~]# 

[Ve0]

Enable the mound firewall service

mound alrady enabled. Firewall I can not enable, because I have oVirt host installed on this server.

Code: Select all

[root@vm ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@vm ~]# 

[hunter86_bg]

Usually when you deploy the RHEV it switches back to the old good iptables/ip6tables, but it could be switched to firewalld if needed. As far as I remember ovirt uses NFSv3 by default. Which version are you using.
If NFSv3 is used - have you defined static ports for the NFS server ,as you might need static ports for the iptables.

[Ve0]

Usually when you deploy the RHEV it switches back to the old good iptables/ip6tables, but it could be switched to firewalld if needed. As far as I remember ovirt uses NFSv3 by default. Which version are you using.
If NFSv3 is used - have you defined static ports for the NFS server ,as you might need static ports for the iptables.

I use NFSv4, it works for me with oVirt 4.1, all ports is fixed in config file.

Code: Select all

[root@vm etc]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54321
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:54322
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sunrpc
ACCEPT     udp  --  anywhere             anywhere             udp dpt:sunrpc
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     udp  --  anywhere             anywhere             udp dpt:snmp
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:websm
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:16514
ACCEPT     tcp  --  anywhere             anywhere             multiport dports rockwell-csp2
ACCEPT     tcp  --  anywhere             anywhere             multiport dports rfb:6923
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 49152:49216
ACCEPT     udp  --  anywhere             anywhere             udp dpt:6081
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:6081
[root@vm etc]# systemctl status iptables
● iptables.service - IPv4 firewall with iptables
   Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2017-04-28 11:32:56 PDT; 2 weeks 0 days ago
 Main PID: 861 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/iptables.service

Apr 28 11:32:56 vm.dazab.local systemd[1]: Starting IPv4 firewall with iptables...
Apr 28 11:32:56 vm.dazab.local iptables.init[861]: iptables: Applying firewall rules: [  OK  ]
Apr 28 11:32:56 vm.dazab.local systemd[1]: Started IPv4 firewall with iptables.
[root@vm etc]#

Add couple rules for NFS:

Code: Select all

iptables -I INPUT -p tcp -m multiport --dport nfs -j ACCEPT
iptables -I INPUT -p ump -m multiport --dport nfs -j ACCEPT

Now if I do map from web server I see all ports:

Code: Select all

[root@web ~]# nmap 192.168.0.99

Starting Nmap 7.40 ( https://nmap.org ) at 2017-05-12 15:36 PDT
Nmap scan report for vm.dazab.local (192.168.0.99)
Host is up (0.000092s latency).
Not shown: 914 filtered ports, 77 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
111/tcp  open  rpcbind
2049/tcp open  nfs
5900/tcp open  vnc
5901/tcp open  vnc-1
5902/tcp open  vnc-2
5903/tcp open  vnc-3
5904/tcp open  unknown
5906/tcp open  unknown
MAC Address: 00:1E:67:14:F3:19 (Intel Corporate)

Nmap done: 1 IP address (1 host up) scanned in 3.67 seconds
[root@web ~]# rpcinfo -p 192.168.0.99
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    2   udp  20048  mountd
    100005    2   tcp  20048  mountd
    100024    1   udp  34321  status
    100024    1   tcp  53508  status
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049  nfs_acl
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    3   udp   2049  nfs_acl
    100021    1   udp  45861  nlockmgr
    100021    3   udp  45861  nlockmgr
    100021    4   udp  45861  nlockmgr
    100021    1   tcp  39327  nlockmgr
    100021    3   tcp  39327  nlockmgr
    100021    4   tcp  39327  nlockmgr
[root@web ~]# showmount -e 192.168.0.99
rpc mount export: RPC: Unable to receive; errno = No route to host
[root@web ~]#

But mount command works fine. Why?

[hunter86_bg]

It looks like your NFS do not support version 3.showmount only works for nfs v3.
What is the output from :

Code: Select all

cat /proc/fs/nfsd/versions

[Ve0]

So far:

Code: Select all

[root@vm ~]# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2
[root@vm ~]#

[hunter86_bg]

OK, so NFSv3 is working. In order to test if it's a firewall or service issue - just drop the firewall down (service iptables stop ) and then try again. If you don't see an error - then you have to recheck all static ports and all firewall rules in the iptables.

[Ve0]

OK, so NFSv3 is working. In order to test if it's a firewall or service issue - just drop the firewall down (service IPtables stop ) and then try again. If you don't see an error - then you have to recheck all static ports and all firewall rules in the IPtables.

I can not drop down firewall. At least 8 VM will stop working. I think the problem in firewall. It is a big deal to set up firewall on hosted engine) But that's ok. Thank you so much for helping.

It's not the end... I have another weird problem. But I will create a new topic.