[WORKEDAROUND] Is it possible to make a DHCP request over a different interface (IB->GbE)?

Issues related to configuring your network
Post Reply
-javier
Posts: 5
Joined: 2017/05/23 08:42:53

[WORKEDAROUND] Is it possible to make a DHCP request over a different interface (IB->GbE)?

Post by -javier » 2017/05/26 11:35:00

Hi all:

I have four computers n0-n3 interconnected with GbE (n0 is frontend) and the "compute" nodes n1-n3 are now also being interconnected with Infiniband (n0 has no IB adapter). DHCP is very convenient for kickstart install (n1-n3 have the same kickstart file, no need to customize network-scripts) and I'd like to also assign IPoIB addresses to "ib1"-"ib3" using DHCP. Is it possible to do that over the GbE switch? n0 has no IB HCA.

That is, I'd like to have

Code: Select all

# ypcat hosts |sort |uniq
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
150.XX.XX.XX  frontend frontend.dom.es
192.168.1.10    n0 nodo0 nodo0.cluster
192.168.1.11    n1 nodo1 nodo1.cluster
192.168.1.12    n2 nodo2 nodo2.cluster
192.168.1.13    n3 nodo3 nodo3.cluster
192.168.2.11    ib1 iband1 iband1.cluster
192.168.2.12    ib2 iband2 iband2.cluster
192.168.2.13    ib3 iband3 iband3.cluster
In the frontend I have

Code: Select all

# ip addr
1: lo: ...
2: enp17s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0a:5e:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 150.XX.XX.XX/23 brd 150.XX.XX.255 scope global enp17s0
       valid_lft forever preferred_lft forever
3: enp13s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1d:7d:XX.XX.XX brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.10/24 brd 192.168.1.255 scope global enp13s0
       valid_lft forever preferred_lft forever
In the nodes I have

Code: Select all

# ip addr
1: lo: ...
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:24:8c:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.11/24 brd 192.168.1.255 scope global dynamic enp6s0
       valid_lft 32919sec preferred_lft 32919sec
3: qlib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 65520 qdisc pfifo_fast state DOWN qlen 256
    link/infiniband 80:00:00:03:fe:80:00:00:00:00:00:00:00:11:75:00:00:XX:XX:XX brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
and

Code: Select all

# cat ifcfg-qlib0
DEVICE=qlib0
TYPE=InfiniBand
ONBOOT=yes
HWADDR=80:00:00:03:fe:80:00:00:00:00:00:00:00:11:75:00:00:XX:XX:XX
BOOTPROTO=dhcp
#BOOTPROTO=none
#IPADDR=192.168.2.11
#PREFIX=24
#NETWORK=192.168.2.0
#BROADCAST=192.168.2.255
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
MTU=65520
CONNECTED_MODE=yes
NAME=ipoib0
I think the Example 10.4 in
https://access.redhat.com/documentation ... erver.html
...would work for me (this was my translation)...

Code: Select all

subnet 192.168.1.0 netmask 255.255.255.0 {
        option routers          n0;
        ...
        filename                "/kickstart/ks.cfg";
}
subnet 192.168.2.0 netmask 255.255.255.0 {
}
host n1 {
        hardware ethernet       00:24:8C:XX:XX:XX;
        option dhcp-client-identifier=80:00:00:03:fe:80:00:00:00:00:00:00:00:11:75:00:00:XX:XX:XX;
        fixed-address           n1,ib1;
}
...if I had an IB adapter on n0. Of course I can get IPoIB working by leaving the BOOTPROTO=none in the ifcfg-* file as offered in
https://access.redhat.com/documentation ... _line.html
but getting the IPoIB address from DHCPd in n0 (via GbE) would allow me to have all the Ethers and HCAs in a single file (dhcpd.conf) and the same kickstart file for all n1-n3 (except a systemctl enable opensm in n1).

I have been reading man dhcpd[.conf], dhclient, dhcrelay, and learning the parameters used by dhclient on CentOS. Reading files /var/lib/NetworkManager/dhclient-*.conf I feel rather "intimidated" :-) (as when reading Procedure 21.1.3 in
https://access.redhat.com/documentation ... setup.html
which by the way I have used for PXE kickstart), and I thought to myself better not to meddle in the affairs of wizards.

I think I want a dhcrelay _before_ the DHCPDISCOVER goes out the HCA, and maybe it can be done with a mix of NetworkManager and dhclient -sf magic in /var/lib/NetworkManager/*conf and/or /usr/libexec/nm-dhcp-helper. Is it possible? Or maybe I have misunderstood it all and it simply can't be done...

Thanks in advance for any ideas
Last edited by -javier on 2017/06/01 16:59:18, edited 1 time in total.

User avatar
jlehtone
Posts: 4523
Joined: 2007/12/11 08:17:33
Location: Finland

Re: Is it possible to make a DHCP request over a different interface (IB->GbE)?

Post by jlehtone » 2017/05/26 14:05:43

I don't think that one connection can use an unrelated connection.

You could creata a static config for the IB during kickstart post-script using the data that the GbE has got from DHCP.

-javier
Posts: 5
Joined: 2017/05/23 08:42:53

Re: Is it possible to make a DHCP request over a different interface (IB->GbE)?

Post by -javier » 2017/05/26 18:08:01

:oops: Hmm, yes, that's much easier. I should have thought it twice before reading the dhcp stuff :D
Thanks a lot!

CaViCcHi
Posts: 68
Joined: 2012/04/18 17:03:15
Contact:

Re: Is it possible to make a DHCP request over a different interface (IB->GbE)?

Post by CaViCcHi » 2017/05/26 18:58:46

I'm not sure that's what you mean but, if you speak of client

dhclient [interface]

does request dhcp over through that interface.

if you mean server, then dhcpd handles that when you, in your configuration file specify the different subnets. so it matches the subnet X with the interface that has that subnet... and son on.

-javier
Posts: 5
Joined: 2017/05/23 08:42:53

Re: Is it possible to make a DHCP request over a different interface (IB->GbE)?

Post by -javier » 2017/05/29 18:59:12

Thanks, sorry for the delay in replying.
My problem is I have no IB adapter in server n0.
I do have Ethernet adapter in n0. Ethernet cards in n1-3 get correctly dhcp-configured. For instance, in n1 there is a dhclient process

Code: Select all

/sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-enp6s0.pid -lf /var/lib/NetworkManager/dhclient-<4B>-<2B>-<2B>-<2B>-<6B>-enp6s0.lease -cf /var/lib/NetworkManager/dhclient-enp6s0.conf enp6s0
I am not too confident it gets started from /etc/sysconfig/network-scripts/ifup-eth (the argument list doesn't really match DHCLIENTARGS there, I'm missing -H, -pf before -lf...), but anyways enp6s0 in n1 is configured as desired (192.168.1.11).

I can read in /var/log/messages

Code: Select all

<date time> n1 NetworkManager[825]: <info>  [xx.0264] dhcp4 (enp6s0): activation: beginning transaction (timeout in 45 seconds)
<date time> n1 NetworkManager[825]: <info>  [xx.0817] dhcp4 (enp6s0): dhclient started with pid 927
<date time> n1 dhclient[927]: DHCPDISCOVER on enp6s0 to 255.255.255.255 port 67 interval 5 (xid=0xXXXXXXXX)
<date time> n1 dhclient[927]: DHCPREQUEST on enp6s0 to 255.255.255.255 port 67 (xid=0xXXXXXXXX)
<date time> n1 dhclient[927]: DHCPOFFER from 192.168.1.10
<date time> n1 dhclient[927]: DHCPACK from 192.168.1.10 (xid=0xXXXXXXXX)
<date time> n1 NetworkManager[825]: <info>  [xx.2290] dhcp4 (enp6s0):   address 192.168.1.11
<date time> n1 NetworkManager[825]: <info>  [xx.2291] dhcp4 (enp6s0):   plen 24 (255.255.255.0)
<date time> n1 NetworkManager[825]: <info>  [xx.2291] dhcp4 (enp6s0):   gateway 192.168.1.10
<date time> n1 NetworkManager[825]: <info>  [xx.2291] dhcp4 (enp6s0):   server identifier 192.168.1.10
<date time> n1 NetworkManager[825]: <info>  [xx.2291] dhcp4 (enp6s0):   lease time 43200
<date time> n1 NetworkManager[825]: <info>  [xx.2291] dhcp4 (enp6s0):   hostname 'n1'
<date time> n1 NetworkManager[825]: <info>  [xx.2291] dhcp4 (enp6s0):   nameserver '150.X.X.X
Then I learned there exists dhcrelay, and thought that if requests can be forwarded, maybe they also can be "redirected", i.e. configure dhclient for the Infiniband device (qlib0) to send the DHCPREQUEST through the ethernet adapter (enp6s0, subnet 192.168.1.0) to get a lease on address 192.168.2.11 for qlib0... not even showing the DHCPREQUEST to the infiniband switch, or even with the IB switch turned off.

I must acknowledge that jlehtone's solution is what I should have tried from the beginning, since I want IPoIB 192.168.2.1x to always be in node nx (IP 192.168.1.1x). ifconfig enp6s0 | grep inet |tr –s \ | cut –d\ -f3 | sed s/\\.1\\./\\.2\\./ might be used to construct the desired IPoIB from the IP assigned by dhcp to the Ether iface.

Sorry for the mess, and thanks anyway for taking the time to reply!

Post Reply