Configuration of infiniband (mlx4_core) after update

[alberto.trj]

Hi,

It is a known issue that the mlx4_core module on CentOS 6 defaults the limit of registrable memory to 32 GB, which can be too low for some systems and make MPI jobs to hang:
https://access.redhat.com/solutions/451223
http://community.mellanox.com/docs/DOC-1120
http://www.open-mpi.org/faq/?category=o ... cked-pages

Adding

Code: Select all

options mlx4_core log_num_mtt=22

to /etc/modprobe.d/mlx4.conf solved the problem and everything was running fine.

However, after updating my system from CentOS 6.5 to 6.6, and libmlx4 from 1.0.5 release 4.el6.1 to 1.0.6 release 6.el6 (both from base repo), the options to the mlx4_core module are no longer read, it seems.

Before the update:

Code: Select all

[root@fisica-03 ~]# cat /etc/modprobe.d/mlx4.conf
options mlx4_core log_num_mtt=22
[root@fisica-03 ~]# cat /sys/module/mlx4_core/parameters/log_num_mtt
22

After the update:

Code: Select all

[root@fisica-05 ~]# cat /etc/modprobe.d/mlx4.conf
options mlx4_core log_num_mtt=22
[root@fisica-05 ~]# cat /sys/module/mlx4_core/parameters/log_num_mtt 
0

How do I make the configuration options to be effective for mlx4_core?

[outpaddling]

FYI, I'm seeing the same issue and so far haven't been able to find a solution.

[chemal]

Last I heard is you don't have to manually tune this anymore, it works out of the box. Are you having any real problems?

[leereyno]

I'm seeing the same problem and it is causing trouble in that OpenMPI is not happy.

I've found a work-around by adding the following to /etc/rc.d/rc.local:

chmod 644 /sys/module/mlx4_core/parameters/log_num_mtt
chmod 644 /sys/module/mlx4_core/parameters/log_mtts_per_seg

echo 24 > /sys/module/mlx4_core/parameters/log_num_mtt
echo 4 > /sys/module/mlx4_core/parameters/log_mtts_per_seg

This should not be necessary however.

[kilian]

Did you try to rename your modprobe.d file to /etc/modprobe.d/mlx4_core.conf instead of mlx4.conf?

[ssgparker]

I'm having the same problem and I've tried all suggestions on this page, but none of them worked for me. Currently using the rc.local workaround.

[chemal]

There is no problem and the worakround is nonsense:

http://www.open-mpi.org/community/lists ... /25090.php