在Ubuntu上用Qemu模拟ARM版本的Fedora39

2024-03-25 12:49 由摩斯电码发表于 #操作系统

参考

Fedora Quick Docs
Fedora Server Documentation
Deploy an ARM64 Fedora VM on your PC: 3 steps
Architectures/AArch64/Install with QEMU
Virtualization – Getting Started
Fedora on Raspberry Pi
Architectures/ARM
ARM Single Board Computer (SBC) Installation
Creating a virtual machine using Fedora Server Edition disk image
8 Linux virsh subcommands for managing VMs on the command line
Use libguestfs to manage virtual machine disk images
虚拟化技术之kvm管理工具virsh常用基础命令

作者

概述

最近在我的家里的T620服务器上折腾Fedora系统时，发现这个系统有很多优点，其中一个就是debuginfod，当你用gdb调试发行版自带的可执行程序时，它可以帮你自动下载符号表和源码包，极其方便，虽然ubuntu22.04也开始提供这个功能，但是尝试了一下，感觉没有fedora好用。我从事的是嵌入式开发，目标机都是ARM64架构的，所以也想在ARM64上把这个功能用起来。

网上看到Fedora支持树莓派4，然后根据这个文章Fedora on Raspberry Pi操作了一下，确实可以。用树莓派官方提供的烧录工具，要写入的OS选择Use custom：

写入的系统也是下面提到的Fedora-Minimal-39-1.5.aarch64.raw.xz

烧写完毕，插到树莓派4B，连上HDMI线和串口，就可以看到启动界面了，测试发现，串口只能输出日志，不能输入，需要连上一个USB键盘到树莓派4B上来输入。

带个树莓派在身边也不太方便，能不能用Qemu模拟一个Fedora的运行环境呢？

步骤

安装虚拟化相关的软件包

sudo apt install virt-manager
sudo apt install qemu-system-aarch64

启动服务并且设置为开机启动

sudo systemctl start libvirtd
sudo systemctl enable libvirtd

下载ARM64版本的fedora39镜像

下面是下载链接：https://alt.fedoraproject.org/alt/ ，因为后面要在x86上通过qemu模拟，选择尽可能小的版本，我用的是“Fedora最小安装”：

下载得到一个xz压缩格式的文件：Fedora-Minimal-39-1.5.aarch64.raw.xz

改造虚拟机镜像

使用7z对这个文件进行解压缩，得到Fedora-Minimal-39-1.5.aarch64.raw

$ 7z x Fedora-Minimal-39-1.5.aarch64.raw.xz
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,12 CPUs Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (906EA),ASM,AES-NI)

Scanning the drive for archives:
1 file, 798690336 bytes (762 MiB)

Extracting archive: ../Fedora-Minimal-39-1.5.aarch64.raw.xz
--
Path = ../Fedora-Minimal-39-1.5.aarch64.raw.xz
Type = xz
Physical Size = 798690336
Method = LZMA2:23 CRC64
Streams = 1
Blocks = 256

Everything is Ok                        

Size:       6442450944
Compressed: 798690336

解压得到的这个文件本身就是一个虚拟机镜像，可以作为直接作为虚拟机镜像运行，可以看一下这个虚拟磁盘的信息：


# 找一个空闲的loop设备
$ sudo losetup -f
/dev/loop19

# 绑定
$ sudo losetup /dev/loop19 ./Fedora-Minimal-39-1.5.aarch64.raw

# 用fdisk查看分区信息
$ sudo fdisk /dev/loop19

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/loop19: 6 GiB, 6442450944 bytes, 12582912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdfc90da8

Device        Boot   Start      End Sectors  Size Id Type
/dev/loop19p1 *       2048  1230847 1228800  600M  6 FAT16
/dev/loop19p2      1230848  3327999 2097152    1G 83 Linux
/dev/loop19p3      3328000 12582911 9254912  4.4G 83 Linux

# 解绑
$ sudo losetup -d /dev/loop19

可以看到，这个磁盘有6GB，太小了，而且是raw格式的，不能打快照，而且太占空间，下面对这个镜像进行改造：

改造成qcow2格式的
将磁盘大小改造为100G

创建一个空的qcow2镜像

$ qemu-img create -f qcow2 fedora39_arm64.qcow2 100G
Formatting 'fedora39_arm64.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=107374182400 lazy_refcounts=off refcount_bits=16

$ ls -lh
total 6.1G
-rw-r--r-- 1 pengdl pengdl 194K 3月  27 14:31 fedora39_arm64.qcow2
-rw-rw-r-- 1 pengdl pengdl 6.0G 3月  27 10:52 Fedora-Minimal-39-1.5.aarch64.raw

绑定到nbd设备

首先需要先加载nbd模块：

$ sudo modprobe nbd

绑定：

sudo qemu-nbd -c /dev/nbd0 ./fedora39_arm64.qcow2

拷贝

$ sudo dd if=./Fedora-Minimal-39-1.5.aarch64.raw of=/dev/nbd0
12582912+0 records in
12582912+0 records out
6442450944 bytes (6.4 GB, 6.0 GiB) copied, 175.658 s, 36.7 MB/s

拷贝完毕，执行一下sync，确保数据落盘，此时也可以打个快照，如果后面扩容出问题，还可以利用快照进行恢复。

对最后一个分区进行扩大

$ sudo fdisk /dev/nbd0

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/nbd0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdfc90da8

Device      Boot   Start      End Sectors  Size Id Type
/dev/nbd0p1 *       2048  1230847 1228800  600M  6 FAT16
/dev/nbd0p2      1230848  3327999 2097152    1G 83 Linux
/dev/nbd0p3      3328000 12582911 9254912  4.4G 83 Linux

# 删除最后一个分区3
Command (m for help): d 
Partition number (1-3, default 3): 3

Partition 3 has been deleted.

# 创建新分区，用默认设置
Command (m for help): n
Partition type
   p   primary (2 primary, 0 extended, 2 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (3,4, default 3): 3
First sector (3328000-209715199, default 3328000): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (3328000-209715199, default 209715199): 

Created a new partition 3 of type 'Linux' and of size 98.4 GiB.
Partition #3 contains a ext4 signature.

# 这里比较重要，因为我们还需要保留原来分区3的内容，这里选择N
Do you want to remove the signature? [Y]es/[N]o: N

# 查看分区是否修改成功
Command (m for help): p
Disk /dev/nbd0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdfc90da8

Device      Boot   Start       End   Sectors  Size Id Type
/dev/nbd0p1 *       2048   1230847   1228800  600M  6 FAT16
/dev/nbd0p2      1230848   3327999   2097152    1G 83 Linux
/dev/nbd0p3      3328000 209715199 206387200 98.4G 83 Linux


# 将修改写入磁盘
Command (m for help): w

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

分区扩大完毕后，文件系统也需要进行调整，否则文件系统还是原来的大小。

检查文件系统错误(否则resize2fs会报错)

$ sudo e2fsck -f /dev/nbd0p3 
e2fsck 1.45.5 (07-Jan-2020)
/dev/nbd0p3 has unsupported feature(s): FEATURE_C12
e2fsck: Get a newer version of e2fsck!

/dev/nbd0p3: ********** WARNING: Filesystem still has errors **********

报错了，提示说我们这个文件系统开启了比较新的特性，当前这个版本的e2fsck不支持，需要使用新版本的工具。这个工具的源码在：https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/

下载后使用下面的命令配置、编译和安装：

$ cd e2fsprogs/
$ ./configure --prefix=`pwd`/install
$ make -j8
$ make install

使用新编译的工具检查文件系统：

$ sudo ./e2fsprogs/install/sbin/e2fsck -f /dev/nbd0p3
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nbd0p3: 41952/289728 files (0.2% non-contiguous), 485030/1156864 blocks

扩大文件系统

$ sudo ./e2fsprogs/install/sbin/resize2fs -f /dev/nbd0p3
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/nbd0p3 to 25798400 (4k) blocks.
The filesystem on /dev/nbd0p3 is now 25798400 (4k) blocks long.

检查文件系统是否扩大成功

$ sudo mount /dev/nbd0p3 ./mnt/                                                                                                                                                 [0/0]
pengdl@pengdl-HP:~/Soft/repeat$ ls mnt/
afs  bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

$ df -hT .
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nbd0p3    ext4   97G  1.8G   91G   2% /home/pengdl/Soft/repeat/mnt

看到上面的结果，表示扩容成功了。

解除跟nbd设备的绑定

$ sudo qemu-nbd -d /dev/nbd0
/dev/nbd0 disconnected

打个快照

$ qemu-img snapshot -c init_version ./fedora39_arm64.qcow2
$ qemu-img snapshot -l ./fedora39_arm64.qcow2
Snapshot list:
ID        TAG               VM SIZE                DATE     VM CLOCK     ICOUNT
1         init_version          0 B 2024-03-27 14:53:53 00:00:00.000          0

启动虚拟机

得到最后改成得到的虚拟机镜像fedora39_arm64.qcow2，然后就可以启动虚拟机了，命令如下：

sudo virt-install \
        --name fedora39_arm64 \
        --virt-type qemu \
        --ram 4096 \
        --disk path=/home/pengdl/Soft/repeat/fedora39_arm64.qcow2,cache=none \
        --nographics \
        --import \
        --arch aarch64 \
        --vcpus 4

等待一会，会出现配置界面，可以在这里创建用户名、密码等：

[  OK  ] Reached target nss-user-lookup.target - User and Group Name Lookups.
         Starting systemd-homed.service - Home Area Manager...
         Starting systemd-logind.service - User Login Management...
[  OK  ] Finished dracut-shutdown.service - Restore /run/initramfs on shutdown.
================================================================================
================================================================================

1) [x] Language settings                 2) [x] Time settings
       (English (United States))                (US/Eastern timezone)
3) [x] Network configuration             4) [x] Root password
       (Connected: enp1s0)                      (Root account is disabled)
5) [ ] User creation
       (No user will be created)


Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 
5
================================================================================
================================================================================

User creation

1) [ ] Create user

Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 
1
================================================================================
================================================================================

User creation

1) [x] Create user
2) Full name
3) User name
4) [x] Use password
5) Password
6) [x] Administrator
7) Groups
   wheel

Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 
3
================================================================================
================================================================================

Enter a new value for 'User name' and press ENTER: 
pengdl                                                                                                                                                                                                             
================================================================================
================================================================================

User creation

1) [x] Create user
2) Full name
3) User name
   pengdl
4) [x] Use password
5) Password
6) [x] Administrator
7) Groups
   wheel

Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 
5
================================================================================
================================================================================

Password: 
Password (confirm): 
================================================================================
================================================================================
Question

The password you have provided is weak: The password fails the dictionary check
- it is too simplistic/systematic
Would you like to use it anyway?


Please respond 'yes' or 'no': 
yes
================================================================================
================================================================================

User creation

1) [x] Create user
2) Full name
3) User name
   pengdl
4) [x] Use password
5) Password
   Password set.
6) [x] Administrator
7) Groups
   wheel

Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 
c
================================================================================
================================================================================

1) [x] Language settings                 2) [x] Time settings
       (English (United States))                (US/Eastern timezone)
3) [x] Network configuration             4) [x] Root password
       (Connected: enp1s0)                      (Root account is disabled)
5) [x] User creation
       (Administrator pengdl will be
       created)


Please make a selection from the above ['c' to continue, 'q' to quit, 'r' to
refresh]: 
c
[  OK  ] Finished initial-setup.service - Initial Setup configuration program.
[  OK  ] Reached target getty-pre.target - Preparation for Logins.
[  OK  ] Started [email protected] - Getty on tty1.
[  OK  ] Started [email protected] - Serial Getty on ttyAMA0.
[  OK  ] Reached target getty.target - Login Prompts.
[  OK  ] Reached target multi-user.target - Multi-User System.
         Starting systemd-update-utmp-runle…- Record Runlevel Change in UTMP...
[  OK  ] Finished systemd-update-utmp-runle…e - Record Runlevel Change in UTMP.

Fedora Linux 39 (Thirty Nine)
Kernel 6.5.6-300.fc39.aarch64 on an aarch64 (ttyAMA0)

fedora login: pengdl
Password: 
[pengdl@fedora ~]$

确认系统信息

# 发行版
[pengdl@fedora ~]$ cat /etc/redhat-release 
Fedora release 39 (Thirty Nine)

# 内核版本
[pengdl@fedora ~]$ cat /proc/version 
Linux version 6.5.6-300.fc39.aarch64 (mockbuild@f67da981c3f1401b99e509796f7601de) (gcc (GCC) 13.2.1 20230918 (Red Hat 13.2.1-3), GNU ld version 2.40-13.fc39) #1 SMP PREEMPT_DYNAMIC Fri Oct  6 19:36:57 UTC 2023

# 文件系统挂载信息
[pengdl@fedora ~]$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/vda3      ext4       97G  1.8G   91G   2% /
devtmpfs       devtmpfs  4.0M     0  4.0M   0% /dev
tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
efivarfs       efivarfs  256K   13K  244K   5% /sys/firmware/efi/efivars
tmpfs          tmpfs     780M  804K  779M   1% /run
tmpfs          tmpfs     2.0G   20K  2.0G   1% /tmp
/dev/vda2      ext4      974M  129M  778M  15% /boot
/dev/vda1      vfat      599M   35M  564M   6% /boot/efi
tmpfs          tmpfs     390M  4.0K  390M   1% /run/user/1000

# 磁盘分区
[pengdl@fedora ~]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
zram0  251:0    0  3.8G  0 disk [SWAP]
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0  600M  0 part /boot/efi
├─vda2 252:2    0    1G  0 part /boot
└─vda3 252:3    0 98.4G  0 part /

# CPU信息
[pengdl@fedora ~]$ lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A57
    Model:               0
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r1p0
    BogoMIPS:            125.00
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-3
Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Vulnerable
  Srbds:                 Not affected
  Tsx async abort:       Not affected

关闭

$ sudo poweroff
...
[  OK  ] Stopped systemd-remount-fs.service…mount Root and Kernel File Systems.
[  OK  ] Stopped systemd-fsck-root.service - File System Check on Root Device.
[  OK  ] Stopped systemd-tmpfiles-setup-dev…ic Device Nodes in /dev gracefully.
         Stopping systemd-zram-setup@zram0.…vice - Create swap on /dev/zram0...
[  OK  ] Stopped [email protected] - Create swap on /dev/zram0.
[  OK  ] Removed slice system-systemd\x2dzr…- Slice /system/systemd-zram-setup.
[  OK  ] Stopped lvm2-monitor.service - Mon…using dmeventd or progress polling.
[  OK  ] Reached target shutdown.target - System Shutdown.
[  OK  ] Reached target final.target - Late Shutdown Services.
[  OK  ] Finished systemd-poweroff.service - System Power Off.
[  OK  ] Reached target poweroff.target - System Power Off.
[  670.116621] reboot: Power down

Domain creation completed.
You can restart your domain by running:
  virsh --connect qemu:///system start fedora39_arm64