QLogic Support
 
 
 
Skip Navigation

Announcements

Dear Customers,


Cavium is in the process of systems integration with the QLogic systems into one combined system for all products. Please bear with us during this process.


For warranty/service verification, please provide your product serial number(s)when contacting support. Serial numbers are located on product bar code labels or from management tools.


Ask Search:
Customer Support Article

RDMA Over Converged Ethernet (RoCE) Install For CentOS And RHEL

  • May 22, 2017
  • ·
  • Like
  • 1

Summary

RDMA Install for CentOS and RHEL

Question

How is the RoCE driver installed on CentOS or Red Hat Enterprise Linux?

Answer

  1. Remove inbox modules
    1. # rmmod qede
    2. # rmmod qed
  2. Install RDMA packages
    1. # yum -y groupinstall "Infiniband Support"
    2. # yum install perftest tcl tcl-devel tk zlib-devel libibverbs libibverbs-devel
  1. Create /etc/udev/rules.d/40-rdma.rules
    1. # vi /etc/udev/rules.d/40-rdma.rules
    2. Insert the following lines
      1. KERNEL=="umad*", NAME="infiniband/%k"
      2. KERNEL=="issm*", NAME="infiniband/%k"
      3. KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
      4. KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
      5. KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
      6. KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"
  2. Add the following lines to the /etc/security/limits.conf file:
    1. * soft memlock unlimited
    2. * hard memlock unlimited
    3. root soft memlock unlimited
    4. root hard memlock unlimited
  3. Logout
    1. # exit
  4. Log back in as root and run the ulimit command – output should be “unlimited”
    1. # ulimit -l
    2. Unlimited
  5. Reboot the server
    1. # reboot
  6. Download and unzip the Driver Source
  7. Untar and install the QLogic driver package for FastlinQ
    1. # tar -xvjf fastlinq-8.15.7.0.tar.bz2
    2. # cd fastlinq-8.15.7.0/
    3. # make clean
    4. # make install
  8. Configure and install userlib for qedr
    1. # cd libqedr-8.15.2.0/
    2. # ./configure --prefix=/usr --libdir=${exec_prefix}/lib --sysconfdir=/etc
    3. # make clean
    4. # make
    5. # make install
  9. Remove existing driver modules
    1. # rmmod qede
    2. # rmmod qed
  10. Install new driver modules
    1. # modprobe -v qedr
      1. Or you can install individually
      2. # modprobe qed
      3. # modprobe qede
      4. # modprobe qedr
  11. Load the RDMA modules (you must perform this step every time you reboot the system)
    1. # modprobe rdma_cm
    2. # modprobe ib_uverbs
    3. # modprobe rdma_ucm
    4. # modprobe ib_ucm
    5. # modprobe ib_umad
    6. # modprobe ib_ipoib
    7. # modprobe ib_iser
  12. Verify modules loaded
    1. # lsmod | grep qed
      1. qedr                  112759  0
      2. qede                  121767  1 qedr
      3. qed                   697989  2 qede,qedr
      4. ib_core               210381  14 qedr,rdma_cm,ib_cm,iw_cm,rpcrdma,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert
  13. Verify the driver version
    1. #modinfo qed
      1. filename: /lib/modules/3.10.0-514.el7.x86_64/extra/qlgc-fastlinq/qed.ko
      2. firmware: qed/qed_init_values_zipped-8.15.3.0.bin
      3. version: 8.15.7.0
      4. license: GPL
      5. description: QLogic FastLinQ 4xxxx Core Module
  14. Verify RoCE functions found
    1. dmesg | grep qedr
      1. [ 1379.599056] ib_srpt srpt_add_one(qedr0)failed.
      2. [ 1379.599086] qedr: discovered and registered 1 RoCE funcs
  15. Type the ibv_devinfo command.
    1. # ibv_devinfo
      1. # ibv_devinfo
      2. hca_id: qedr0
      3. transport:              InfiniBand (0)
      4. fw_ver:                 8.15.3.0
      5. node_guid:              020e:1eff:fed7:1469
      6. sys_image_guid:         020e:1eff:fed7:1469
      7. vendor_id:              0x1077
      8. vendor_part_id:         5700
      9. hw_ver:                 0x0
      10. phys_port_cnt:          1
      11. port:                   1
      12. state:                  PORT_DOWN (1)
      13. max_mtu:                4096 (5)
      14. active_mtu:             1024 (3)
      15. sm_lid:                 0
      16. port_lid:               0
      17. port_lmc:               0x00
      18. link_layer:             Ethernet

Attachment