Proxmox 7.4 使用vgpu_unlock,为GTX1060开启vGPU支持
本文在 2021年发布的博客《Proxmox 5.4使用vgpu_unlock,为GTX1060开启vGPU支持》,介绍了 Proxmox VE 5.4 上部署vGPU unlock 的操作步骤。
后续有发布了在 Proxmox VE 7.x 上支持 vGPU的博客《Proxmox 7.2 部署 DoraCloud桌面云,支持vGPU》, 实现了通过 3个脚本完成 vGPU的配置。
本文在上述两个文章的基础上,实现通过4个脚本,实现利用消费级显卡实现vGPU。包含了对 Proxmox VE的更新,启用iommu,安装 nvidia grid 驱动,以及加载 vgpu_unlock。
准备工作:
硬件: 一台服务器,配置有ssd硬盘, 安装有GTX 1060 显卡。
软件: 安装了 Proxmox VE 7.4 。
网络: 网络中可以访问互联网。
步骤1:换源,更新
curl -o- http://www1.deskpool.com:9000/software/gpu01.sh |bash
上面的脚本的内容,为了方便阅读,贴在这里
#!/bin/sh # Author: DoraCloud Technology Ltd.co # # Date: 2022/05/07 # # DoraCloud for Proxmox Enable vGPU # Phase 1: update source to mirrors.ustc.edu.cn cp /etc/apt/sources.list /etc/apt/sources.list.backup sed -i 's|^deb http://ftp.debian.org|deb https://mirrors.ustc.edu.cn|g' /etc/apt/sources.list sed -i 's|^deb http://security.debian.org|deb https://mirrors.ustc.edu.cn/debian-security|g' /etc/apt/sources.list mv /etc/apt/sources.list.d/pve-enterprise.list /etc/apt/sources.list.d/pve-enterprise.list.bak CODENAME=`cat /etc/os-release |grep PRETTY_NAME |cut -f 2 -d "(" |cut -f 1 -d ")"` #echo "deb https://mirrors.nju.edu.cn/proxmox/debian/pve $CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list #echo "deb https://mirrors.ustc.edu.cn/proxmox/debian $CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list wget https://mirrors.ustc.edu.cn/proxmox/debian/proxmox-release-$CODENAME.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-$CODENAME.gpg #更新 apt update && apt install pve-kernel-5.15 pve-headers-5.15 dkms mdevctl -y
步骤2: 启用iommu
curl -o- http://www1.deskpool.com:9000/software/gpu02.sh |bash
上面的脚本的内容,为了方便阅读,贴在下方
#!/bin/sh # Author: DoraCloud Technology Ltd.co # # Date: 2022/05/07 # # Enable IO-MMU on PVE Server # 复制如下脚本,启用IO-MMU echo "" echo "********************************************" echo "*** Enable IO-MMU on proxmox host ***" echo "********************************************" # /etc/default/grub 的GRUB_CMDLINE_LINUX_DEFAULT,增加 intel_iommu=on iommu=pt sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"/g' /etc/default/grub echo "" echo " Update grub .... " update-grub # 加载 vfio vfio_iommu_type1 vfio_pci vfio_virqfd 4个Modules echo vfio >> /etc/modules echo vfio_iommu_type1 >> /etc/modules echo vfio_pci >> /etc/modules echo vfio_virqfd >> /etc/modules echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf echo "blacklist nouveau" >>/etc/modprobe.d/disable-nouveau.conf echo "options nouveau modeset=0" >>/etc/modprobe.d/disable-nouveau.conf update-initramfs -u echo "" echo " Proxmox host reboot ............." reboot
步骤3:安装 NVIDIA GRID 13.6的驱动
curl -o- http://www1.deskpool.com:9000/software/gpu03e.sh |bash
上面的脚本的内容,请自己阅读。 这里选择相对稳定的 13.6 版本。
#!/bin/sh # Author: DoraCloud Technology Ltd.co # # Date: 2022/05/07 # # Install NVIDIA Linux vGPU Driver 470.161.02 echo "" echo "********************************************************" echo "*** Install NVIDIA Grid vGPU Driver 13.6 V470.161.02 ***" echo "********************************************************" echo " Downloading NVIDIA driver " wget http://www1.deskpool.com:9000/software/NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run chmod +x NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run echo "" echo " Extracting driver .... " ./NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm.run -x cd NVIDIA-Linux-x86_64-470.161.02-vgpu-kvm/ echo "" echo " Patching NVIDIA driver " sed -i 's|.open = nv_vgpu_vfio_open|.open_device = nv_vgpu_vfio_open|g' kernel/nvidia-vgpu-vfio/nvidia-vgpu-vfio.c sed -i 's|.release = nv_vgpu_vfio_close|.close_device = nv_vgpu_vfio_close|g' kernel/nvidia-vgpu-vfio/nvidia-vgpu-vfio.c echo "" echo " Installing driver .... " ./nvidia-installer -dkms -s echo "" echo " deamon-reload && rebooting ...." systemctl daemon-reload reboot
步骤4:运行 vgpu_unlock 和 vgpu_unlock-rs
curl -o- http://www1.deskpool.com:9000/software/gpu03u.sh |bash
上面的脚本的内容,请自己阅读。
#!/bin/sh # Author: DoraCloud Technology Ltd.co # # Date: 2022/05/07 # # Unlock NVIDIA GTX 1060 echo "" echo "********************************************************" echo "*** Unlock NVIDIA GTX1060 ***" echo "********************************************************" echo "" echo "" echo " Download vgpu_lock " echo "" apt install git-core -y git clone https://gitee.com/deskpool/vgpu_unlock.git chmod -R +x vgpu_unlock echo "" echo "" echo "********************************************************" echo " Apply vgpu_unlock to 470.161.02" echo "" sed -i 's/#include "nv-time.h"/#include "nv-time.h"\n\n#include "\/root\/vgpu_unlock\/vgpu_unlock_hooks.c"/g' /usr/src/nvidia-470.161.02/nvidia/os-interface.c echo "ldflags-y += -T /root/vgpu_unlock/kern.ld" >> /usr/src/nvidia-470.161.02/nvidia/nvidia.Kbuild dkms remove -m nvidia -v 470.161.02 dkms install -m nvidia -v 470.161.02 echo "" echo "" echo "********************************************************" echo " Install RUST " echo "" #设置rustup的source为 USTC export RUSTUP_DIST_SERVER=https://mirrors.ustc.edu.cn/rust-static export RUSTUP_UPDATE_ROOT=https://mirrors.ustc.edu.cn/rust-static/rustup curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y #wget -qO- https://cdn.jsdelivr.net/gh/rust-lang-nursery/rustup.rs/rustup-init.sh |sh -s -- -y echo "" echo "" echo "********************************************************" echo " Config cargo " echo "" #加载cargo的环境变量 source ~/.cargo/env #设置cargo的源镜像为 USTC cat >~/.cargo/config <<EOF [source.crates-io] replace-with = 'ustc' [source.ustc] registry = "git://mirrors.ustc.edu.cn/crates.io-index" EOF echo "" echo "" echo "********************************************************" echo " Building vgpu_unlock-rs " echo "" #下载 vgpu_unlock-rs 项目 git clone https://gitee.com/deskpool/vgpu_unlock-rs cd vgpu_unlock-rs/ cargo build --release echo "" echo "" echo "********************************************************" echo " Create vgpu.service && vgpu-mgr.service " echo "" #nvidia-vgpud 的unlock 服务 mkdir /etc/systemd/system/nvidia-vgpud.service.d cat >/etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf <<EOF [Service] Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so EOF #nvidia-vgpu-mgr 的unlock服务 mkdir /etc/systemd/system/nvidia-vgpu-mgr.service.d cat >/etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf <<EOF [Service] Environment=LD_PRELOAD=/root/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so EOF #描述文件,控制GPU的显示配置,以及cuda mkdir /etc/vgpu_unlock cat >/etc/vgpu_unlock/profile_override.toml <<EOF [profile.nvidia-55] num_displays = 1 display_width = 1920 display_height = 1080 max_pixels = 2073600 cuda_enabled = 1 frl_enabled = 0 EOF echo "" echo "" echo "********************************************************" echo " Reboot Proxmox VE ..... " echo " run mdevctl types to check vGPU is installed. " echo "" sync reboot
里面的内容包括了安装 vgpu_unlock ,安装 rust 环境, 下载和编译 vgpu_unlock-rs, 配置 nvidia-vgpu 和nvidia-vgpu-mgr的服务,
生成vgpu_unlock 显卡配置文件 profile_override.toml 等内容。
全部执行完毕后,系统重启之后,运行 mdevctl types ,可以检查到 vGPU的类型已经生成。
root@pve03:~# mdevctl types 0000:01:00.0 nvidia-156 Available instances: 12 Device API: vfio-pci Name: GRID P40-2B Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12 nvidia-215 Available instances: 12 Device API: vfio-pci Name: GRID P40-2B4 Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12 nvidia-241 Available instances: 24 Device API: vfio-pci Name: GRID P40-1B4 Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24 nvidia-283 Available instances: 6 Device API: vfio-pci Name: GRID P40-4C Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2160, max_instance=6 nvidia-284 Available instances: 4 Device API: vfio-pci Name: GRID P40-6C Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2160, max_instance=4 nvidia-285 Available instances: 3 Device API: vfio-pci Name: GRID P40-8C Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2160, max_instance=3 nvidia-286 Available instances: 2 Device API: vfio-pci Name: GRID P40-12C Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2 nvidia-287 Available instances: 1 Device API: vfio-pci Name: GRID P40-24C Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2160, max_instance=1 nvidia-46 Available instances: 24 Device API: vfio-pci Name: GRID P40-1Q Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24 nvidia-47 Available instances: 12 Device API: vfio-pci Name: GRID P40-2Q Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12 nvidia-48 Available instances: 8 Device API: vfio-pci Name: GRID P40-3Q Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8 nvidia-49 Available instances: 6 Device API: vfio-pci Name: GRID P40-4Q Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6 nvidia-50 Available instances: 4 Device API: vfio-pci Name: GRID P40-6Q Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4 nvidia-51 Available instances: 3 Device API: vfio-pci Name: GRID P40-8Q Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3 nvidia-52 Available instances: 2 Device API: vfio-pci Name: GRID P40-12Q Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2 nvidia-53 Available instances: 1 Device API: vfio-pci Name: GRID P40-24Q Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1 nvidia-54 Available instances: 24 Device API: vfio-pci Name: GRID P40-1A Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24 nvidia-55 Available instances: 12 Device API: vfio-pci Name: GRID P40-2A Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12 nvidia-56 Available instances: 8 Device API: vfio-pci Name: GRID P40-3A Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8 nvidia-57 Available instances: 6 Device API: vfio-pci Name: GRID P40-4A Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6 nvidia-58 Available instances: 4 Device API: vfio-pci Name: GRID P40-6A Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4 nvidia-59 Available instances: 3 Device API: vfio-pci Name: GRID P40-8A Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3 nvidia-60 Available instances: 2 Device API: vfio-pci Name: GRID P40-12A Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2 nvidia-61 Available instances: 1 Device API: vfio-pci Name: GRID P40-24A Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1 nvidia-62 Available instances: 24 Device API: vfio-pci Name: GRID P40-1B Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
剩下的工作是部署 GuestOS,并安装 NVIDIA GRID 驱动。 这里推荐部署 DoraCloud,下载 win10LTSC2021V3 模板。 内置了GRID 13.6的驱动。
部署的方法,参考 《Proxmox 7.2 部署 DoraCloud桌面云,支持vGPU》 ,以及 《ShareStation工作站虚拟化部署 (doracloud.cn) 》
最后分享以下, GTX 1060 虚拟化之后,vGPU 运行《原神》游戏的效果。