### https://docs.nvidia.com/networking/display/initialdpumgmtandprovisioning/prerequisites+for+initial+bluefield+dpu+deployment #download https://linux.mellanox.com/public/repo/doca/2.7.0/rhel9.3/x86_64/ # 查看 # sudo update-pciids lspci -d 15b3: -vvv | grep --color Width #uninstall for f in $(rpm -qa |grep -i doca ) ; do yum -y remove $f; done yum autoremove #install wget https://www.mellanox.com/downloads/DOCA/DOCA_v2.7.0/host/doca-host-2.7.0-204000_24.04_rhel93.x86_64.rpm rpm -Uvh doca-host-2.7.0-204000_24.04_rhel93.x86_64.rpm yum makecache sudo yum install doca-ofed -y sudo systemctl restart rshim sudo systemctl status rshim #验证驱动是否成功挂载并识别 ls /dev/ | grep rshim [root@localhost rshim0]# cat /dev/rshim0/misc DISPLAY_LEVEL 0 (0:basic, 1:advanced, 2:log) BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap) BOOT_TIMEOUT 150 (seconds) DROP_MODE 0 (0:normal, 1:drop) SW_RESET 0 (1: reset) DEV_NAME pcie-0000:c1:00.2 DEV_INFO BlueField-3(Rev 1) OPN_STR N/A UP_TIME 39(s) SECURE_NIC_MODE 0 (0:no, 1:yes) [root@localhost rshim0]# # https://docs.nvidia.com/networking/display/bluefielddpuosv385/deploying+dpu+os+using+bfb+from+host#src-71012946_DeployingDPUOSUsingBFBfromHost-InstallingUbuntuonBlueField #Installing Ubuntu on BlueField [root@localhost ~]# openssl passwd -1 Password: Verifying - Password: $1$u0QDSqzU$T2grDRRksYPuaaNR1KIAC. [root@localhost ~]#cat bf.cfg ubuntu_PASSWORD='$1$u0QDSqzU$T2grDRRksYPuaaNR1KIAC.' #BFB Installation https://developer.nvidia.com/networking/doca# bfb-install -h bfb-install --bfb .bfb --config bf.cfg --rshim rshim0 [root@localhost ~]# bfb-install --bfb DOCA_2.6.0_BSP_4.6.0_Ubuntu_22.04-5.24-01.prod.bfb --config bf.cfg --rshim rshim0 Checking if local host has root access... Checking if rshim driver is running locally... Warn: 'pv' command not found. Continue without showing BFB progress. Pushing bfb + cfg Collecting BlueField booting status. Press Ctrl+C to stop… INFO[PSC]: PSC BL1 START INFO[BL2]: start INFO[BL2]: boot mode (rshim) INFO[BL2]: VDDQ adjustment complete INFO[BL2]: VDDQ: 1120 mV INFO[BL2]: DDR POST passed INFO[BL2]: UEFI loaded INFO[BL31]: start INFO[BL31]: lifecycle GA Secured INFO[BL31]: VDD: 878 mV INFO[BL31]: runtime INFO[UEFI]: eMMC init INFO[UEFI]: eMMC probed INFO[UEFI]: UPVS valid INFO[UEFI]: PMI: updates started INFO[UEFI]: PMI: total updates: 1 INFO[UEFI]: PMI: updates completed, status 0 INFO[UEFI]: PCIe enum start INFO[UEFI]: PCIe enum end INFO[UEFI]: UEFI Secure Boot (enabled) INFO[UEFI]: Redfish enabled INFO[UEFI]: exit Boot Service INFO[MISC]: : Found bf.cfg INFO[MISC]: : Ubuntu installation started INFO[MISC]: Installing OS image INFO[MISC]: : Changing the default password for user ubuntu INFO[MISC]: : Ubuntu installation completed INFO[MISC]: : Installation finished [root@localhost ~]# cat /dev/rshim0/misc DISPLAY_LEVEL 2 (0:basic, 1:advanced, 2:log) BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap) BOOT_TIMEOUT 150 (seconds) DROP_MODE 0 (0:normal, 1:drop) SW_RESET 0 (1: reset) DEV_NAME pcie-0000:c1:00.2 DEV_INFO BlueField-3(Rev 1) OPN_STR N/A UP_TIME 516(s) SECURE_NIC_MODE 0 (0:no, 1:yes) --------------------------------------- Log Messages --------------------------------------- INFO[PSC]: PSC BL1 START INFO[BL2]: start INFO[BL2]: boot mode (emmc) INFO[BL2]: VDDQ: 1120 mV INFO[BL2]: DDR POST passed INFO[BL2]: UEFI loaded INFO[BL31]: start INFO[BL31]: lifecycle GA Secured INFO[BL31]: VDD: 881 mV INFO[BL31]: runtime INFO[UEFI]: eMMC init INFO[UEFI]: eMMC probed INFO[UEFI]: UPVS valid INFO[UEFI]: PCIe enum start INFO[UEFI]: PCIe enum end INFO[UEFI]: PMI: updates started INFO[UEFI]: PMI: total updates: 1 INFO[UEFI]: PMI: updates completed, status 0 INFO[UEFI]: PMI: updates started INFO[UEFI]: PMI: total updates: 7 INFO[UEFI]: PMI: updates completed, status 0 INFO[UEFI]: UEFI Secure Boot (enabled) INFO[UEFI]: Redfish enabled INFO[UEFI]: DPU-BMC RF credentials not found WARN[UEFI]: Var reclaim WARN[UEFI]: Var reclaim done INFO[UEFI]: exit Boot Service INFO[MISC]: Linux up INFO[MISC]: DPU is ready INFO[MISC]: : DPU is ready [root@localhost ~]# sudo ip addr add 192.168.100.1/24 dev tmfifo_net0 ssh ubuntu@192.168.100.2 Password: [root@localhost ~]# ip a ... 6: tmfifo_net0: mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000 link/ether 00:1a:ca:ff:ff:02 brd ff:ff:ff:ff:ff:ff inet 192.168.100.1/24 scope global tmfifo_net0 valid_lft forever preferred_lft forever inet6 fe80::21a:caff:feff:ff02/64 scope link valid_lft forever preferred_lft forever [root@localhost ~]# ssh ubuntu@192.168.100.2 The authenticity of host '192.168.100.2 (192.168.100.2)' can't be established. ED25519 key fingerprint is SHA256:dGq1zSsbxnz3843FehCBkKWz6RtGKHMCN4lG3WgGORY. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.100.2' (ED25519) to the list of known hosts. ubuntu@192.168.100.2's password: Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-1035-bluefield aarch64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro System information as of Tue May 14 07:46:05 UTC 2024 System load: 0.12646484375 Processes: 310 Usage of /: 4.8% of 116.76GB Users logged in: 0 Memory usage: 5% IPv4 address for oob_net0: 192.168.1.174 Swap usage: 0% IPv4 address for tmfifo_net0: 192.168.100.2 Temperature: 57.1 C Expanded Security Maintenance for Applications is not enabled. 0 updates can be applied immediately. Enable ESM Apps to receive additional future security updates. See https://ubuntu.com/esm or run: sudo pro status The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. To run a command as administrator (user "root"), use "sudo ". See "man sudo_root" for details. ubuntu@localhost:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS mmcblk0 179:0 0 38.8G 0 disk ├─mmcblk0p1 179:1 0 50M 0 part └─mmcblk0p2 179:2 0 38.8G 0 part mmcblk0boot0 179:8 0 31.5M 1 disk mmcblk0boot1 179:16 0 31.5M 1 disk nvme0n1 259:0 0 119.2G 0 disk ├─nvme0n1p1 259:1 0 50M 0 part /boot/efi └─nvme0n1p2 259:2 0 119.2G 0 part / ubuntu@localhost:~$ lscpu Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Cortex-A78AE Model: 1 Thread(s) per core: 1 Core(s) per socket: 16 Socket(s): 1 Stepping: r0p1 BogoMIPS: 660.47 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm paca pacg Caches (sum of all): L1d: 1 MiB (16 instances) L1i: 1 MiB (16 instances) L2: 8 MiB (16 instances) L3: 16 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-15 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; CSV2, BHB Srbds: Not affected Tsx async abort: Not affected ubuntu@localhost:~$ #Firmware Upgrade sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update ubuntu@localhost:/opt/mellanox/mlnx-fw-updater$ sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update Initializing... Attempting to perform Firmware update... Querying Mellanox devices firmware ... Device #1: ---------- Device Type: BlueField3 Part Number: 900-9D3B6-00CV-A_Ax Description: NVIDIA BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled PSID: MT_0000000884 PCI Device Name: 03:00.0 Base MAC: a088c2a973a0 Versions: Current Available FW 32.38.3056 32.40.1000 PXE 3.7.0201 N/A UEFI 14.31.0022 N/A UEFI Virtio blk 22.4.0012 N/A UEFI Virtio net 21.4.0012 N/A Status: Update required --------- Found 1 device(s) requiring firmware update... Device #1: Updating FW ... FSMST_INITIALIZE - OK Writing Boot image component - OK Done Restart needed for updates to take effect. Log File: /tmp/wcX14MLb2s Real log file: /tmp/mlnx_fw_update.log ubuntu@localhost:/opt/mellanox/mlnx-fw-updater$