Diskless netboot server
Diskless netboot server
The DBC-CloudCyberCafe project uses iPXE to boot, uses iSCSI for diskless services, uses lvm2 for storage management, uses thrift for API interfaces, and uses plotly to provide console interfaces. Together they provide customers with diskless network boot services, so deploy the diskless netboot server is to deploy these modules separately. The server system uses ubuntu 20.04.
iPXE
PXE was proposed by Intel to boot from the network card. Obtain IP through DHCP and obtain startup files through TFTP. iPXE is an enhanced and extended version of PXE, which supports multiple acquisition methods such as HTTP, so DHCP, TFTP and HTTP services need to be deployed.
DHCP
Dynamic Host Configuration Protocol DHCP is a standard protocol defined by RFC 1541 (replaced by RFC 2131), which allows servers to dynamically assign IP addresses and configuration information to clients. And the PXE service needs to use the bootfile field of the DHCP protocol to tell the machine the PXE startup file. If there is already a DHCP service (DHCP address assignment provided by the router) in Internet cafes and other places, it is recommended to use dnsmasq as a DHCP proxy. If there is no DHCP service, it is recommended to directly use isc-dhcp-server to configure your own subnet. The DBC-CloudCyberCafe project uses the dnsmasq service by default, because many customer environments already have routers, and only need dnsmasq to act as a DHCP protocol agent to provide fields such as bootfile. At this time, it is only necessary to install dnsmasq and successfully start dnsmasq.service. The specific configuration will be automatically generated by the console program of DBC-CloudCyberCafe project.
# install dnsmasq
sudo apt install dnsmasq
# view dnsmasq service status
systemctl status dnsmasq.service
注意！
The default systemd-resolved of the ubuntu system will listen to port 53 for DNS resolution. Starting the dnsmasq.service service may fail. There are many solutions. Here is a reference: use the command systemd-resolve --interface=eno1 --set-dns=223.5.5.5 Manually set the dns address for the network card eno1 to 223.5.5.5, then restart the systemd-resolved.service service, and then start the dnsmasq.service service to succeed.
In addition, if you need to install isc-dhcp-server, please refer to the following steps.
# install isc-dhcp-server
sudo apt install isc-dhcp-server
# view isc-dhcp-server service status
systemctl status isc-dhcp-server.service
注意！
To use the isc-dhcp-server service, you need to set a specific network card in the /etc/default/isc-dhcp-server file, and specify which network card to enable the DHCP service for.
TFTP
# install TFTP
sudo apt install tftpd-hpa
# view tftpd-hpa service status
systemctl status tftpd-hpa.service
tftpd-hpa uses the /srv/tftp folder as the storage directory by default. You need to put undionly.kpxe and ipxe.efi in this directory. These two files can use the files provided by the ipxe official website. We recommend using custom scripts to compile them file yourself, and how to compile iPXE will be introduced later.
The download address of the startup file provided by the iPXE official website is as follows：
- undionly.kpxe: https://boot.ipxe.org/undionly.kpxe
- ipxe.efi: https://boot.ipxe.org/ipxe.efi
HTTP
iPXE can use a more stable and reliable HTTP protocol to download the required files. You can use apache or nginx to build an HTTP server. Take nginx as an example below：
# install nginx，refer to http://nginx.org/en/linux_packages.html#Ubuntu
sudo apt install nginx
Add the following server block to the http block in the /etc/nginx/nginx.conf file：
server {
listen 8080;
root /var/www/file;
location / {
autoindex on;# show directory
autoindex_exact_size on;# show file size
autoindex_localtime on;# show file time
}
}
Finally, restart nginx to build the /var/www/file directory into a file server, and use the 8080 port url to access it in the browser.
The /var/www/file directory structure is as follows：
dbtu@dbtu:/var/www/file$ tree
.
├── ipxe
│ ├── boot.ipxe
│ ├── boot.ipxe.cfg
│ ├── cfg
│ │ ├── 404.html
│ │ ├── mac-0050562ca04b.ipxe.cfg
│ │ ├── mac-40b0767ee231.ipxe.cfg
│ │ └── mac-88aedd0508fa.ipxe.cfg
│ └── wimboot
├── netboot
│ ├── ldlinux.c32 -> ubuntu-installer/amd64/boot-screens/ldlinux.c32
│ ├── netboot.tar.gz
│ ├── pxelinux.0 -> ubuntu-installer/amd64/pxelinux.0
│ ├── pxelinux.cfg -> ubuntu-installer/amd64/pxelinux.cfg
│ ├── ubuntu-installer
│ │ └── amd64
│ │ ├── boot-screens
│ │ │ ├── adtxt.cfg
│ │ │ ├── exithelp.cfg
│ │ │ ├── f10.txt
│ │ │ ├── f1.txt
│ │ │ ├── f2.txt
│ │ │ ├── f3.txt
│ │ │ ├── f4.txt
│ │ │ ├── f5.txt
│ │ │ ├── f6.txt
│ │ │ ├── f7.txt
│ │ │ ├── f8.txt
│ │ │ ├── f9.txt
│ │ │ ├── ldlinux.c32
│ │ │ ├── libcom32.c32
│ │ │ ├── libutil.c32
│ │ │ ├── menu.cfg
│ │ │ ├── prompt.cfg
│ │ │ ├── rqtxt.cfg
│ │ │ ├── splash.png
│ │ │ ├── stdmenu.cfg
│ │ │ ├── syslinux.cfg
│ │ │ ├── txt.cfg
│ │ │ └── vesamenu.c32
│ │ ├── initrd.gz
│ │ ├── linux
│ │ ├── pxelinux.0
│ │ └── pxelinux.cfg
│ │ └── default -> ../boot-screens/syslinux.cfg
│ └── version.info
├── ubuntu
│ └── preseed.cfg
└── winpe
├── amd64
│ ├── fwfiles
│ │ ├── efisys.bin
│ │ └── etfsboot.com
│ ├── media
│ │ ├── bg-bg
│ │ │ └── bootmgr.efi.mui
│ │ ├── Boot
│ │ │ ├── BCD
│ │ │ ├── bg-bg
│ │ │ │ └── bootmgr.exe.mui
│ │ │ ├── bootfix.bin
│ │ │ ├── boot.sdi
│ │ │ ├── en-us
│ │ │ │ ├── bootmgr.exe.mui
│ │ │ │ └── memtest.exe.mui
│ │ │ ├── Fonts
│ │ │ │ ├── chs_boot.ttf
│ │ │ │ ├── cht_boot.ttf
│ │ │ │ ├── jpn_boot.ttf
│ │ │ │ ├── kor_boot.ttf
│ │ │ │ ├── malgun_boot.ttf
│ │ │ │ ├── malgunn_boot.ttf
│ │ │ │ ├── meiryo_boot.ttf
│ │ │ │ ├── meiryon_boot.ttf
│ │ │ │ ├── msjh_boot.ttf
│ │ │ │ ├── msjhn_boot.ttf
│ │ │ │ ├── msyh_boot.ttf
│ │ │ │ ├── msyhn_boot.ttf
│ │ │ │ ├── segmono_boot.ttf
│ │ │ │ ├── segoen_slboot.ttf
│ │ │ │ ├── segoe_slboot.ttf
│ │ │ │ └── wgl4_boot.ttf
│ │ │ ├── memtest.exe
│ │ │ ├── Resources
│ │ │ │ └── bootres.dll
│ │ │ ├── zh-cn
│ │ │ │ ├── bootmgr.exe.mui
│ │ │ │ └── memtest.exe.mui
│ │ │ └── zh-tw
│ │ │ ├── bootmgr.exe.mui
│ │ │ └── memtest.exe.mui
│ │ ├── bootmgr
│ │ ├── bootmgr.efi
│ │ ├── EFI
│ │ │ ├── Boot
│ │ │ │ ├── bootx64.efi
│ │ │ │ └── en-us
│ │ │ │ └── bootx64.efi.mui
│ │ │ └── Microsoft
│ │ │ └── Boot
│ │ │ ├── BCD
│ │ │ ├── en-us
│ │ │ │ └── memtest.efi.mui
│ │ │ ├── Fonts
│ │ │ │ ├── chs_boot.ttf
│ │ │ │ ├── cht_boot.ttf
│ │ │ │ ├── jpn_boot.ttf
│ │ │ │ ├── kor_boot.ttf
│ │ │ │ ├── malgun_boot.ttf
│ │ │ │ ├── meiryo_boot.ttf
│ │ │ │ ├── msjh_boot.ttf
│ │ │ │ ├── msyh_boot.ttf
│ │ │ │ ├── segmono_boot.ttf
│ │ │ │ ├── segoe_slboot.ttf
│ │ │ │ └── wgl4_boot.ttf
│ │ │ ├── memtest.efi
│ │ │ ├── Resources
│ │ │ │ └── bootres.dll
│ │ │ ├── zh-cn
│ │ │ │ └── memtest.efi.mui
│ │ │ └── zh-tw
│ │ │ └── memtest.efi.mui
│ │ ├── en-us
│ │ │ └── bootmgr.efi.mui
│ │ ├── sources
│ │ │ └── boot.wim
│ └── mount
├── lightningWinPE
│ ├── boot
│ │ ├── bcd
│ │ └── boot.sdi
│ ├── bootmgr
│ ├── bootmgr.efi
│ ├── efi
│ │ ├── boot
│ │ │ └── bootx64.efi
│ │ └── microsoft
│ │ └── boot
│ │ └── bcd
│ └── sources
│ └── BOOT.WIM
├── wepe
│ ├── EFI
│ │ ├── BOOT
│ │ │ └── bootx64.efi
│ │ └── MICROSOFT
│ │ └── BOOT
│ │ ├── BCD
│ │ └── FONTS
│ │ └── wgl4_boot.ttf
│ ├── WEIPE
│ └── WEPE
│ ├── B64
│ ├── FONTS
│ │ └── wgl4_boot.ttf
│ ├── MESSAGE
│ ├── PELOAD
│ ├── WEIPE
│ ├── WEPE64
│ ├── WEPE64.WIM
│ ├── WEPE.INI
│ ├── WEPE.SDI
│ └── WEPE.TXT
└── WePE64_V2.2.iso
- The iPXE startup script is stored in the ipxe folder, where mac-000c29c63944.ipxe.cfg is named after the MAC address in all lowercase format without the colon. Each booted machine needs to have a corresponding configuration file, which is automatically configured by the console program.
- The wimboot files are downloaded from https://github.com/ipxe/wimboot/releases.
- The PE file used to install the Windows system is stored under winpe, which is not needed for the time being and is used for development and testing.
- netboot is the file used to install ubuntu on the network. It can be downloaded from Alibaba Cloud or Tsinghua University, reference file download address. If the customer needs the ubuntu system image, he needs to download this file.
- ubuntu/preseed.cfg is the unattended configuration file used by network installations of ubuntu.
The content of the ipxe/boot.ipxe file is as follows:
#!ipxe
chain --autofree boot.ipxe.cfg
chain --replace cfg/mac-${mac:hexraw}.ipxe.cfg
The content of the ipxe/boot.ipxe.cfg file is as follows：
#!ipxe
set iscsi-server 192.168.1.159
set base-url http://192.168.1.159:8080
set menu-timeout 5000
Among them, the IP address of iscsi-server is the IP address of the diskless netboot server, and base-url is the url of the HTTP service. Please set these contents according to the actual situation.
iPXE file configuration
The process of compiling iPXE can refer to: https://ipxe.org/download
git clone https://github.com/ipxe/ipxe.git
cd ipxe/src
Create script.ipxe in the ipxe/src folder, the content of the file is:
#!ipxe
dhcp
chain --autofree http://192.168.1.2:8080/ipxe/boot.ipxe
Here 192.168.1.2 is the IP address of the diskless netboot server, and port 8080 is the port of the HTTP service configured by nginx, please fill in according to their actual configuration.
Then use script.ipxe to compile the iPXE boot file, and the boot.ipxe configuration provided by the HTTP service can be automatically loaded after the PXE boot is started. Use the following command to compile:
make bin-x86_64-pcbios/undionly.kpxe EMBED=script.ipxe
make bin-x86_64-efi/ipxe.efi EMBED=script.ipxe
Copy the compiled undionly.kpxe and ipxe.efi files to the TFTP service directory.
sudo cp bin-x86_64-pcbios/undionly.kpxe /srv/tftp/
sudo cp bin-x86_64-efi/ipxe.efi /srv/tftp/
iSCSI
iSCSI (Internet Small Computer System Interface, pronounced /ˈаɪskʌzi/), Internet Small Computer System Interface, also known as IP-SAN, is a storage technology based on the Internet and the SCSI-3 protocol, proposed by the IETF and published in 2003. It became an official standard on February 11, 2010.
iSCSI uses TCP/IP ports 860 and 3260 as communication channels. By using the iSCSI protocol to exchange SCSI commands between two computers, the computer can emulate a SAN as a local storage device through a high-speed LAN hub.
Essentially, iSCSI lets two hosts negotiate with each other over an IP network and then exchange SCSI commands. In this way, iSCSI emulates a common high-performance local storage bus with a wide area network, creating a storage area network (SAN). Unlike some SAN protocols, iSCSI does not require dedicated cabling; it can run over existing switching and IP infrastructure.
For computers with a network interface device (NIC) that supports network boot, an additional DHCP server can be configured to assist with iSCSI boot. In this case, the NIC looks for a DHCP server that provides a PXE or BOOTP boot image. The DHCP server will provide the corresponding iSCSI boot target device/volume information according to the MAC address of the boot network card, and then the computer can start the process of remotely booting from iSCSI.
iSCSI can be divided into a server and a client. A scsi target needs to be installed on the server to share storage devices, and an iscsi initiator needs to be installed on the client to connect to the target. The device shared by the target needs to be mounted to the local initiator, and can be partitioned and formatted.
iSCSI client
The iSCSI initiator is the one that initiates the I/O operation; it needs to request the remote block device through the discovery process; it can establish a persistent connection with the target; in the Linux system, the open-iscsi software package can be used to simulate the implementation, and in Windows 10 and later system, we can use the built-in iSCSI initiator to achieve.
iSCSI server side
The iSCSI target is the executor of the I/O operation; one or more block devices need to be exported for the initiator (initiator); two target tools can be used in the Linux system, namely tgt and targetcli, here we first introduce TGT Method, TGT is that Fujita Tomonori introduced SCSI Target Framework (STGT/TGT) into the Linux kernel at the end of 2006. It has a library in the kernel, which can assist the kernel to control the target driver. TGT is the iscsi target implemented in user mode, and all target processing is performed in user space. At the end of 2010, the LIO (Linux-IO) project was selected to replace TGT as the iscsi target implemented in the kernel state. When LIO was chosen to replace TGT, its implementation has been tweaked to allow TGT user empty modules to continue running, so the TGT community supports the inclusion of LIO in the kernel. Before Linux kernel 2.6.38 it was TGT.
Linux-IO
In the Linux kernel (after linux 2.6.38), Linux-IO Target uses software to implement various SCSI Targets. It supports all popular storage protocols in SAN technology, including Fiber Channel (Qlogic, linux3.5), FCoE (linux3. 0), iSCSI (linux 3.1), iSER (Mellanox InfiniBand, linux3.10), SRP (Mellanox InfiniBand, linux3.3), USB, etc. At the same time, it can also generate simulated SCSI devices for this machine, and provide virtual machines based on Virtio's SCSI device. Linux-IO Target enables users to use a relatively cheap Linux system to realize various functions of SCSI and SAN without purchasing expensive professional equipment.
Linux-IO Target realizes the simulation of Target in the kernel state, and adopts a more "modern" sysfs-based method for configuration management, providing a friendly user-mode management tool. From version 2.6.38 of the kernel, the Linux kernel includes the relevant modules of Linux-IO Target. In the user mode, targetcli and rtslib are packaged in all major releases. The targetcli program is used for configuration and management, and rtslib provides a Python programming interface.
Install the iSCSI server side using the following command:
sudo apt install targetcli-fb open-iscsi
lvm2
LVM (Logical Volume Manager) means logical volume management. It is a mechanism for managing disk partitions in the Linux environment. LVM is a logical layer built on hard disks and partitions to improve the flexibility of disk partition management.
- PV(physical volume): usually a partition, or an entire physical hard disk, or a raid device.
- VG(volume group): A collection of multiple physical volumes, created on top of the physical volume. When the volume group is created, physical volumes can be dynamically added to the volume group. There can be only one volume group or multiple volume groups in a logical volume system project.
- LV(logical volume): Created on top of volume groups, logical volumes can also be dynamically expanded or reduced in size. Unallocated volume group space in the volume group can be used to create new logical volumes. Multiple logical volumes can belong to the same volume group or to different volume groups.
The command to install lvm2 is as follows:
sudo apt install lvm2
Here it is recommended to use the pvcreate command to initialize the safe and fast RAID device as a physical volume, use the vgcreate command to create a volume group on the physical volume, and then fill in the volume group name in the settings of the diskless console. The desktop program will automatically create a logical volume on the volume group, create a diskless mirror on the logical volume, use the mirror file as the back-end storage of iSCSI, and finally connect the iSCSI back-end with the iPXE boot program to attach it as a network disk, and we can start the system in the disk and save data to the disk like a local disk.
thrift
The DBC-CloudCyberCafe project uses the thrift rpc framework to provide an API interface to facilitate third-party programs (mainly dbc programs) to set the machine to enter the Windows or Ubuntu system, modify the system login password, and close the diskless client. Therefore, the diskless netboot server needs to install the necessary thrift tools.
sudo apt install thrift-compiler python3-pip
sudo pip3 install thrift
diskless console
The DBC-CloudCyberCafe project uses plotly to implement a diskless console to facilitate customer managers to add machines, bind MAC addresses and IP addresses, and modify machine startup items.
Because the operation of lvm2 and iSCSI requires the root user authority of the diskless netboot server, the diskless console may wait for the root user password to be entered, resulting in many subsequent operations failing, so it is recommended that the user who sets the server do not need to enter the password when executing the sudo command by adding the content dbc ALL=(ALL) NOPASSWD:ALL in /etc/sudoers. The contents of the /etc/sudoers file are as follows:
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults env_reset
Defaults mail_badpass
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
# Host alias specification
# User alias specification
# Cmnd alias specification
# User privilege specification
root ALL=(ALL:ALL) ALL
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) ALL
dbc ALL=(ALL) NOPASSWD:ALL
# See sudoers(5) for more information on "#include" directives:
#includedir /etc/sudoers.d
The command to deploy the diskless console from source code is as follows:
# Download the DBC-CloudCyberCafe project code
git clone https://github.com/DeepBrainChain/DBC-CloudCyberCafe.git
cd DBC-CloudCyberCafe/preset/
thrift --gen py preset.thrift
cd ../plotly/
sudo python3 home.py
As long as you execute the sudo python3 home.py command in the DBC-CloudCyberCafe/plotly/ directory of the project code, you can start the console program of the diskless service, and then enter http://localhost:8050/ in the browser to access the console. Don't forget to replace localhost with the IP address of the diskless netboot server.
After opening the diskless console in the browser, you need to fill in some settings on the Setting page.
You can also use a packaged executable program to deploy a diskless console, see https://github.com/DeepBrainChain/DBC-CloudCyberCafe/releases .
- Storage
- Volume Group: Fill in the volume group created in the lvm2 step above.
- DHCP
- network name: Network or subnet name, default
dbc. - interface: Network card device name, the network card connected in a LAN, such as
eno1. - subnet: Network segment, IP address segment, such as
192.168.1.0. - subnet mask: Subnet mask, such as
255.255.255.0. - range: Starting IP address and ending IP address range, for example
192.168.1.100-192.168.1.200. - routers: Gateway IP address, such as
192.168.1.1. - dns servers: DNS server address, such as
114.114.114.114and Alibaba Cloud public DNS223.5.5.5. - broadcast address: Broadcast address, such as
192.168.1.255. - filename: The address of the script file loaded after iPXE starts, for example
http://192.168.1.2:8080/ipxe/boot.ipxe. - next server: The IP address of the TFTP server deployed above, such as
192.168.1.2.
- network name: Network or subnet name, default
- HTTP
- root path: The storage directory of the HTTP file service deployed above, the console program will generate the configuration corresponding to the MAC address of the machine under its ipxe/cfg folder.
- http ip:port: The IP address and port of the HTTP service deployed above, such as
http://192.168.1.2:8080.
- iSCSI
- iscsi server: IP address of the iSCSI server deployed above. For example
192.168.1.2. - initiator iqn: A public iSCSI initiator name, such as
iqn.2022-10.org.dbc.iscsi:global.client. - target prefix: Public iSCSI target name prefix, eg
iqn.2022-10.org.dbc.iscsi.
- iscsi server: IP address of the iSCSI server deployed above. For example
注意！
- Theoretically, services such as TFTP, HTTP, and iSCSI can be deployed on different servers, so the settings distinguish many server addresses. It is recommended to use one server and use a secure RAID for lvm2.
- The filename in DHCP is the script loaded after iPXE starts, if undionly.kpxe and ipxe.efi have been compiled with custom scripts, this item can be ignored.