QLogic Support
 
 
 
Skip Navigation

Announcements

Dear Customers,

Maintenance is scheduled for the QLogic (Cavium) support site.
The customer support log-in page, driver downloads, and license key activation page may be unavailable between Friday, October 27, at 13:00 PDT, and will remain offline until Monday, October 30, at 08:00 PDT.
We apologize for any inconvenience. You may still open a support case at globalsupport@cavium.com during this time.



For warranty/service verification, please provide your product serial number(s)when contacting support. Serial numbers are located on product bar code labels or from management tools.


Ask Search:
Customer Support Article

RDMA Over Converged Ethernet (RoCE) Install For CentOS And RHEL

  • May 22, 2017
  • ·
  • Like
  • 1

Summary

RDMA Install for CentOS and RHEL

Question

How is the RoCE driver installed on CentOS or Red Hat Enterprise Linux?

Answer

  1. Remove inbox modules
    1. # rmmod qede
    2. # rmmod qed
  2. Install RDMA packages
    1. # yum -y groupinstall "Infiniband Support"
    2. # yum install perftest tcl tcl-devel tk zlib-devel libibverbs libibverbs-devel
  1. Create /etc/udev/rules.d/40-rdma.rules
    1. # vi /etc/udev/rules.d/40-rdma.rules
    2. Insert the following lines
      1. KERNEL=="umad*", NAME="infiniband/%k"
      2. KERNEL=="issm*", NAME="infiniband/%k"
      3. KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
      4. KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
      5. KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
      6. KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"
  2. Add the following lines to the /etc/security/limits.conf file:
    1. * soft memlock unlimited
    2. * hard memlock unlimited
    3. root soft memlock unlimited
    4. root hard memlock unlimited
  3. Logout
    1. # exit
  4. Log back in as root and run the ulimit command – output should be “unlimited”
    1. # ulimit -l
    2. Unlimited
  5. Reboot the server
    1. # reboot
  6. Download and unzip the Driver Source
  7. Untar and install the QLogic driver package for FastlinQ
    1. # tar -xvjf fastlinq-8.15.7.0.tar.bz2
    2. # cd fastlinq-8.15.7.0/
    3. # make clean
    4. # make install
  8. Configure and install userlib for qedr
    1. # cd libqedr-8.15.2.0/
    2. # ./configure --prefix=/usr --libdir=${exec_prefix}/lib --sysconfdir=/etc
    3. # make clean
    4. # make
    5. # make install
  9. Remove existing driver modules
    1. # rmmod qede
    2. # rmmod qed
  10. Install new driver modules
    1. # modprobe -v qedr
      1. Or you can install individually
      2. # modprobe qed
      3. # modprobe qede
      4. # modprobe qedr
  11. Load the RDMA modules (you must perform this step every time you reboot the system)
    1. # modprobe rdma_cm
    2. # modprobe ib_uverbs
    3. # modprobe rdma_ucm
    4. # modprobe ib_ucm
    5. # modprobe ib_umad
    6. # modprobe ib_ipoib
    7. # modprobe ib_iser
  12. Verify modules loaded
    1. # lsmod | grep qed
      1. qedr                  112759  0
      2. qede                  121767  1 qedr
      3. qed                   697989  2 qede,qedr
      4. ib_core               210381  14 qedr,rdma_cm,ib_cm,iw_cm,rpcrdma,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert
  13. Verify the driver version
    1. #modinfo qed
      1. filename: /lib/modules/3.10.0-514.el7.x86_64/extra/qlgc-fastlinq/qed.ko
      2. firmware: qed/qed_init_values_zipped-8.15.3.0.bin
      3. version: 8.15.7.0
      4. license: GPL
      5. description: QLogic FastLinQ 4xxxx Core Module
  14. Verify RoCE functions found
    1. dmesg | grep qedr
      1. [ 1379.599056] ib_srpt srpt_add_one(qedr0)failed.
      2. [ 1379.599086] qedr: discovered and registered 1 RoCE funcs
  15. Type the ibv_devinfo command.
    1. # ibv_devinfo
      1. # ibv_devinfo
      2. hca_id: qedr0
      3. transport:              InfiniBand (0)
      4. fw_ver:                 8.15.3.0
      5. node_guid:              020e:1eff:fed7:1469
      6. sys_image_guid:         020e:1eff:fed7:1469
      7. vendor_id:              0x1077
      8. vendor_part_id:         5700
      9. hw_ver:                 0x0
      10. phys_port_cnt:          1
      11. port:                   1
      12. state:                  PORT_DOWN (1)
      13. max_mtu:                4096 (5)
      14. active_mtu:             1024 (3)
      15. sm_lid:                 0
      16. port_lid:               0
      17. port_lmc:               0x00
      18. link_layer:             Ethernet

Attachment