GB/T 45401.1-2025Artificial intelligence - Scheduling and cooperation for computing devices - Part 1: Virtualization and scheduling (English PDF)

1 Scope

This document gives the architecture for the virtualization and scheduling of artificial intelligence computing devices, specifies the technical requirements and describes the test methods.

This document applies to the system design, development and testing of the virtualization and scheduling of artificial intelligence computing devices.

2 Normative references

The contents of the following documents constitute indispensable provisions of this document through normative reference in the text. For dated references, only the edition corresponding to that date applies to this document; for undated references, the latest edition, including all amendments, applies to this document.

GB/T 41867 Information technology — Artificial intelligence — Terminology. GB/T 45087-2024 Artificial intelligence — Test methods for the performance of server systems.

3 Terms and definitions

The terms and definitions given in GB/T 41867 and the following apply to this document.

Artificial intelligence computing unit: the smallest set of components needed to carry out an artificial intelligence computing task. Note: an artificial intelligence computing unit is generally packaged in an artificial intelligence accelerator or accelerator card.

Artificial intelligence accelerating processor unit, also artificial intelligence accelerating chip: an integrated circuit component having a computing micro-architecture suited to artificial intelligence algorithms and able to carry out the computing processing of artificial intelligence applications.

Artificial intelligence accelerating card: an expansion acceleration device designed specifically for artificial intelligence computing and conforming to the hardware interface of an artificial intelligence server. Note: artificial intelligence accelerator cards are divided by scenario of use into artificial intelligence training accelerator cards, artificial intelligence inference accelerator cards and others.

Artificial intelligence computing instance: the virtualized object that carries out an artificial intelligence computing task.

Virtualization: a form of resource representation used to express decoupling from the underlying physical resources. [Source: ISO/IEC 17826:2022, 3.55]

Heterogeneous resource pool: an abstract entity formed by the collection of artificial intelligence computing resources of different architectures. Note 1: a heterogeneous resource pool provides a scalable computing architecture, which is favourable to the rational allocation of computing resources and provides guarantees of computing capability, storage, bandwidth and latency for the development and deployment of artificial intelligence application systems in different operating environments, for example cloud, cluster, mobile device and internet of things. Note 2: a heterogeneous resource pool is used to manage and schedule artificial intelligence computing resources so as to meet the needs of different artificial intelligence computing tasks. Note 3: artificial intelligence computing resources include the central processing unit (CPU), the graphics processing unit (GPU), the neural-network processing unit (NPU), the field programmable gate array (FPGA), the digital signal processor (DSP), the application specific integrated circuit (ASIC) and others.

Computing capability: the greatest degree to which a product or system can meet a computing need.

Neural-network processing unit: an integrated circuit component specially optimized in design for neural network computation. Note: this kind of integrated circuit component is good at processing multimedia data such as video, image and speech.

Artificial intelligence computing task: the activity needed to achieve a particular artificial intelligence computing objective. Note: in this document, where no misunderstanding arises from the context, an artificial intelligence computing task generally means an inference task or a training task.

Performance: a characteristic that can be measured when a computing task is being run. Note 1: performance includes both qualitative and quantitative features. Note 2: performance is obtained from the measurement or calculation of one or more parameters, such as energy consumption, traffic, throughput, running time and rate, so as to characterize the behaviour, the features and the efficiency of a technical process running in a given machine. Note 3: in evaluating the performance of an artificial intelligence task, the throughput characteristic is generally used. [Source: ISO/IEC 20000-10:2018, 3.1.16, modified]

Artificial intelligence computing cluster: a collection of artificial intelligence computing functional units following unified control. Note 1: artificial intelligence computing functional units may include artificial intelligence accelerators, artificial intelligence servers, artificial intelligence acceleration modules and others. Note 2: when made up of artificial intelligence servers, an artificial intelligence cluster is also called an artificial intelligence server cluster.

Node: a physical or logical artificial intelligence computing device, connected by a network, that can complete a particular artificial intelligence computing task. [Source: ISO/IEC 14575:2000, 3.2.27, modified]

Scheduling: the process of controlling the place and the time at which the whole or part of a particular task is executed. Note: in this document, place generally means the artificial intelligence computing unit. [Source: ISO/IEC 10164-15:2002, 3.7.4, modified]

Scheduler: the component that carries out scheduling in a system. Note: in this document, the scheduler is used to allocate artificial intelligence computing resources for different computing needs.

Scheduling policy: the policy by which the place and the time of execution of the whole or part of a task is matched in a system so as to complete the scheduling of the task.

Isolation: the condition in which computing instances neither affect one another nor have access to one another in respect of computation and data. Note: computing power isolation means that the computing capability of one computing instance does not affect that of another. [Source: ISO/IEC TS 25052-1:2022, 3.1.5.3, modified]

4 Abbreviated terms

The following abbreviated terms apply to this document. AI, artificial intelligence. API, application programming interface. BAR, base address register. CPU, central processing unit. DDR, double data rate. DMA, direct memory access. FLOPS, floating-point operations per second. FPGA, field programmable gate array. FPS, frames per second. FP16, 16-bit floating-point number. GDDR, graphics DDR SDRAM. GPA, guest physical address. GPU, graphics processing unit. HBM, high bandwidth memory. ID, identity document. INT8, 8-bit integer number. IO, input output. IOMMU, input output memory management unit. IOVA, input output virtual address. NIC, network interface card. NPU, neural-network processing unit. OPS, operations per second. OS, operating system. PCIE, peripheral component interconnect express. QEMU, quick emulator. QoS, quality of service. SR-IOV, single root input output virtualization. VFIO, virtual function input output. VM, virtual machine. VMM, virtual machine manager. VMX, virtual machine extension.

5 Overview

The architecture for the virtualization and scheduling of AI computing devices is given in Figure 1. AI computing virtualization provides a particular form of representation for the physical AI computing resources; the virtualization schemes cover the virtualization of physical AI accelerator cards based on CPU, GPU, NPU, FPGA and the like. The several virtualization schemes form a resource pool through a unified access component, so that the physical AI computing resources are used in a consistent manner. According to the AI task and the state of the resource pool, the scheduler selects a number of virtualized AI computing instances, allocates them and executes the particular task. Operation and maintenance monitoring provides monitoring and control of the AI computing instances, of the physical AI computing resources, that is the AI accelerator cards, and of the AI tasks and their states.

Note 1: the parts shown in dashed boxes do not fall within the scope of standardization of this document. Note 2: the virtualization scheme for the FPGA covers the mixed architecture of FPGA and CPU. Note 3: one AI application is generally broken down into a number of AI computing tasks, which are handed to the scheduler.

6.1 Technical requirements for computing device virtualization — Overview

Computing device virtualization technology is a method of computing resource management which divides, deploys or consolidates different AI accelerator cards containing accelerators of various types, so that they carry out AI computing tasks and the utilization of computing resources is raised.

Note: this document standardizes computing device virtualization technology; it puts no requirements on other physical resources such as network and storage, but it does put requirements on the cooperative interfacing of computing resources with other resources.

6.2 Basic requirements

AI accelerator card virtualization. The virtualization of an AI accelerator card includes the following requirements: a) it shall support the virtualization of a single AI accelerator card into several computing instances; b) it should support the virtualization of several AI accelerator cards into a single computing instance; c) a computing instance shall be able to carry out a computing task in a virtual machine or a container; d) it should support computing power isolation on a single AI accelerator card; e) a computing instance shall be able to be called by a computing device.

Monitoring. The virtualization component or system should carry out the following monitoring of the attributes of computing instances: a) monitoring of resource utilization, including computing resources; b) statistics on the execution latency of operators; c) monitoring of the bandwidth utilization of the computing instance; d) monitoring of the performance fluctuation of the computing instance, see 9.1.4.4; e) statistics on the loss of computing capability of the AI accelerator card after virtualization; f) statistics on the running time of the computing instance.

Isolation. The isolation of the virtualization component or system includes the following requirements. a) It shall support fault isolation, meeting the following: single point fault information within a computing instance is reported only to the virtual machine or container to which it belongs and to the virtualization management system; several computing instances are supported in running without interfering with one another; business faults calling or returning from a particular computing instance are provided; and a computing instance is supported in learning of a single point fault of the physical device on which it depends. b) It shall support performance isolation, that is, the performance with which a particular user uses computing resources shall not fall as a result of being affected by the tasks of other users running concurrently. c) It shall support security isolation, including but not limited to the prevention of unauthorized access and of attack.

Reconfiguration. The virtualization component or system carries out the reconfiguration of computing resources under conditions of multiple users and dynamic tasks, so as to raise the overall performance of the system, meeting the following. a) It shall implement a QoS quality of service mechanism for the independent running of a single AI task. b) It shall implement a mechanism for fairness in obtaining resources for the simultaneous running of several AI tasks. Note: fairness means that different users and programs of the same priority have the same right to use the computing resources. c) It shall implement a resource multiplexing mechanism, such as time-division multiplexing or partition multiplexing, for the dynamic load characteristics or variable resource requirements of AI tasks. d) It should be able to be used together with the following mechanisms or processes, including but not limited to: a high-availability AI computing cluster; load balancing, that is the allocation of particular business, such as network services or network traffic, to several network devices, including servers and firewalls, or to several links; memory multiplexing, by such means as memory ballooning, memory swapping and memory sharing; and the expansion of CPU, memory, AI accelerating processor and the like. e) It should support the querying of interconnection information between AI computing accelerator cards.

Resource abstraction. The resource abstraction of the virtualization and scheduling system meets the following. a) A computing instance shall support the execution of the following computations, including but not limited to: complete convolution computation; complete vector computation; complete scalar computation; and data transfer, exchange and format conversion. b) A computing instance shall contain the following components, which serve as the basis of scheduling: computing components; storage components, such as cache and memory; and data communication components. c) One computing instance may correspond to all or part of the physical resources of one or several AI computing devices. d) Where several computing instances are virtualized, the hardware resources contained in a single AI accelerator card shall be able to be mapped into several AI accelerator cards, including the division of resources such as processor and memory, mapped as different devices and limited by the total number of hardware resources. e) Where events, identifiers and tasks and the related mechanisms are implemented in software, they shall not be limited by the total number of hardware resources.

Coordination. The task abstraction and the resource abstraction of the computation graph of an AI task shall be coordinated, meeting the following. Note: an AI computation graph is represented by a directed graph; a directed graph is made up of vertices and edges, represents the computation flow of a neural network, is a kind of logical flow graph, and is converted into a physical flow graph on the hardware. a) The resource abstraction in the data flow graph of an AI task includes: a vertex in the directed graph represents a computation; an edge in the directed graph represents a data dependency; a data flow connecting line with an output arrow represents writing data into data storage; and a data flow connecting line with an input arrow represents reading data from data storage. b) The virtual device used to execute the data flow graph should be taken into account. c) The computation flow should take account of the following constraints: synchronization of the computation flow, that is, a task in one computation flow depends on a task in another computation flow; and computation flows may be processed in parallel, subject to hardware resource constraints.

Representation of a computing instance. A computing instance shall be represented by the following attributes. a) The specification of the computing instance, comprising: computing capability, such as throughput, effective computing capability and FLOPS; memory space, such as on-chip memory; data transfer rate, such as uplink and downlink bandwidth; and computing precision, such as FP16 and INT8. b) The topology of the computing instance, comprising: the AI accelerator card to which the computing instance belongs; the server to which the accelerator card belongs and its position within the server; and the AI computing cluster to which the server belongs and its position within the AI computing cluster. c) The state of the computing instance, comprising: in use, meaning that the computing instance is currently carrying out a related computing task; available, meaning that the current computing instance in whole or in part is idle; and unavailable, meaning that the current computing instance is already occupied or damaged and cannot carry out the related task.

Management of computing instances. The requirements for the management of computing instances are as follows. a) The AI computing device virtualization system or component shall support the updating of the resource pool in the following circumstances: a new computing instance joins; a computing instance is released; an idle computing instance is occupied; a computing instance enters the waiting queue; a computing instance is damaged; a damaged computing instance is repaired. b) The virtualization of a single AI accelerator card should meet the following functions through interface calls: the AI virtual computing resource created is associated with the parent device to which it belongs, that is the physical AI accelerator on which the AI virtual computing resource sits, and the related queries are provided, including but not limited to a Python programming interface, a sysfs interface and commands; interfaces for the operation and control of computing instances; a writable interface for entering the information needed to create a virtualization node; an interface for obtaining device description information; driver interface information for the virtualized computing node, that is the driver API; and a soft link pointing to each virtualized computing node. c) The AI computing device virtualization system or component shall support the deletion and restoration of computing resources, including the non-physical deletion of computing resources and the restoration of deleted resources, for example the rediscovery of computing resources. d) The AI computing device virtualization system or component shall support the provision of interfaces for obtaining the following instance information: monitoring and statistical information on computing capability, such as the sum of the physical computing capability managed by the resource scheduling layer; information on the AI accelerator card on which the computing instance sits; specification information of the computing instance, including memory and observable bandwidth; and operator latency profiling and latency statistics of the computing instance. e) The AI computing device virtualization system or component shall support the allocation of computing resources, including the allocation of a single AI accelerator and the atomic allocation of several AI accelerators as a whole, that is, either all are allocated successfully or none is. f) The AI computing device virtualization system or component shall support the reclaiming of computing instances, including the reclaiming of a single computing instance at a time and the reclaiming of several computing instances at a time, with the atomicity of the operation guaranteed.

6.3 Extended requirements

CPU virtualization — overview. CPU virtualization is divided into software-based virtualization and hardware-assisted virtualization. The CPU is generally used for inference tasks. A virtual machine may use several CPU computing instances, and these computing instances may come from the virtualization of the same physical CPU or of several physical CPUs. The reference architecture for CPU virtualization is given in A.2 of Annex A.

CPU computing instance. The CPU virtualization system or component shall meet the following. a) The CPU computing instance can carry out machine learning inference tasks. b) Waste of the CPU resources of the host machine can be avoided. c) Dynamic horizontal elastic scaling of CPU computing instances is supported, so that CPU computing instances can be added to a virtual machine. d) Affinity configuration of CPU computing instances is supported. e) The specification of a CPU computing instance is measured by the following indicators: number of processor cores; number of threads per core; the proportion into which the CPU is divided; memory capacity; cache capacity.

GPU virtualization — general. The virtualization of a GPU shall meet the following requirements: a) it shall support the use of computing capability on demand; b) it shall support the dynamic calling and release of computing capability.

GPU computing instance. The GPU virtualization system or component shall meet the following. a) It supports the machine learning algorithms that its non-virtualized version can execute. b) Computing instances are obtained by time-division multiplexing or space-division multiplexing. c) A GPU computing instance contains the following components: computing components, that is the stream processors that execute the computation and carry out integer and floating-point computation; video encoding and decoding components; memory, including but not limited to DDR, GDDR and HBM, with capacity in MB or GB and bandwidth in GB/s; caches at each level, with capacity in MB or GB and bandwidth in GB/s; and shared storage. d) Load balancing is supported. e) The specification of a GPU computing instance shall be measured by the following indicators: computing capability, in units such as OPS, FLOPS, throughput or computing power proportion; video encoding and decoding capability, in units such as one channel of 1080 P at 30 FPS; video memory capacity, in units such as MB or GB; cache capacity, in units such as MB; and network bandwidth between GPUs and packet transmit and receive throughput.

NPU virtualization — general. The virtualization architecture of the NPU is given in Figure 2 and the reference scheme in A.1. The NPU virtualization component or system meets the following. a) It shall be able to select a suitable virtualization and scheduling scheme according to the needs of the computing task, including: virtualizing the AI accelerator card into several NPU computing instances mounted to a virtual machine; virtualizing the AI accelerator card into several NPU computing instances mounted to a container; virtualizing the AI accelerator card into several NPU computing instances mounted to a virtual machine and then, within the virtual machine, mounting the NPU computing instances to containers; and passing the NPU of the AI accelerator card through to a virtual machine, dividing the NPU computing instances within the virtual machine and then mounting them to containers inside the virtual machine. b) It should have an NPU computing QoS control mechanism. c) An NPU instance shall provide the following computing attributes: tensor computing capability, such as the number of tensor computing units; general computing capability such as vector and scalar, for example the number of general computing units and the number of coprocessor cores; and, where a video and image preprocessing module is present, the related computing capability attributes such as the number of video decoder computing cores. d) An NPU instance shall provide memory capacity, in GB, as a storage attribute. e) It should support the dynamic adjustment of the specification of a computing instance according to the amount of computation of the AI task.

NPU computing instance. An NPU computing instance shall meet the following. a) It provides a matrix multiplication computing component that executes convolution and fully connected computation: a computing core that executes matrix computation; buffer matrices that store the intermediate process data and the result data separately; and data registers that hold the process data and the result of matrix operations. b) It provides a vector operation component, so as to execute operations such as pooling, activation functions and batch normalization: a unit that executes vector computation; and the related storage units. c) It provides a computing component for control and scalar operations, comprising: a unit that executes scalar operations; general registers; and special registers. d) It provides a data storage component. e) It provides a data path component, so as to guarantee the supply of data between the computing components and the storage components. f) The NPU computing instance can execute and complete AI inference and training tasks. g) The specification of an NPU computing instance is measured by the following indicators: computing capability, in FLOPS, OPS or throughput; memory capacity, in GB; cache, in KB or MB; and remaining bandwidth, in GB/s.

FPGA virtualization — general. FPGA virtualization is the scheduling and use, in the dimensions of time and space, of the various resources of several FPGAs. The virtualization of an FPGA shall meet the following requirements: a) it shall support deep learning acceleration and have the property of performance isolation; b) it shall support multi-channel computation; c) it shall support FPGA overlay, that is, forming a virtual programmable architecture on top of the FPGA hardware for use by the upper layers; d) it shall support partial reconfiguration, dividing the FPGA into several FPGA computing instances; e) it shall support dynamic FPGA reconfiguration and configuration.

FPGA computing instance. An FPGA computing instance shall meet the following. a) It supports the machine learning tasks that its non-virtualized version can execute. b) It is one or several dynamically reconfigured FPGA regions. c) When dynamic reconfiguration is carried out on a particular FPGA computing instance, the running of the others is not affected. d) It provides a mechanism for dividing the reconfigurable regions, so as to avoid FPGA resources being left empty and wasted. e) The specification of an FPGA computing instance is measured by the following indicators: computing capability, for example OPS or throughput; latency; power consumption; and flexibility.

7.1 Technical requirements for computing resource scheduling — Overview

Task flow framework. The task flow of computing resource scheduling is given in Figure 3 and comprises the following steps: a) the user raises a training or inference request, which is put into the task pool as an AI task; b) the scheduler takes a task from the AI task pool and, according to the specification of the task, that is the computing capability and memory it needs, and the available AI accelerator cards, applies for a computing instance that meets the requirements of the AI task; c) the scheduler monitors all the heterogeneous computing resources and computing instances and completes the scheduling of the task.

Training scenario. In the training scenario the task pool contains training task requests. Where the throughput of the training task is large, the efficiency with which the resources of a single AI accelerator card are used is high. The occupancy of AI accelerator card resources by a single training task covers the following situations: a) running on a single AI accelerator card with all the resources of that card occupied; b) running on a single AI accelerator card with the resources of that card not all occupied; c) running on several AI accelerator cards with the resources of all the accelerator cards occupied; d) running on several AI accelerator cards with idle resources on some of the accelerator cards.

Inference scenario. The occupancy of AI computing units by inference tasks covers the following situations. a) Where the model is small enough to be held in the memory of a single card or a single chip: inference needs heterogeneous AI computing resources, but in most scenarios the throughput is small and resource usage is low; the scheduling of computing resources needs a fine-grained division, for example taking a block or a die within a single card as the scheduling unit, so that several inference tasks share particular computing resources, that is a single-chip multi-task mode; and at deployment the computing resources needed are applied for according to the computing task, inference tasks being arranged on a single chip in preference so that the single chip is fully loaded. Note: fully loaded means that the computing capability of the particular computing device is entirely used for virtualization into computing instances. b) Where the model is too large to be held in the memory of a single card or a single chip: the model has many parameters, a complex structure and a large task throughput, and is split into several sub-models; data movement is fastest within a single chip, and cross-card scheduling affects computing speed; the scheduling of computing resources is allocated by card; the scheduler splits the inference task according to the size of the model; and the same sub-model is arranged in preference on a single AI accelerator card, the remaining sub-models being allocated to the card for computation once the computation is complete.

7.2 Functional requirements

The scheduler shall implement the following functions: a) the scheduling of a whole AI accelerator card, that is the processor; b) the scheduling of pooled multiple AI accelerator cards; c) the scheduling of the computing units on a single AI accelerator card; d) the scheduling of the several computing instances obtained by the virtualization of a single AI accelerator card; e) the scheduling of several computing units across several AI accelerator cards; f) the scheduling of several computing units or several AI accelerator cards, with the atomicity of the scheduling operation guaranteed; g) task splitting, so that where the same task contains several models, the different models can be allocated to different computing instances, with the global utilization of resources kept at its highest; h) model switching, so that on the same computing instance, when one model has finished running, another model can be swapped in and run; i) load balancing, so that where computing nodes remain, tasks can be allocated and excessive load on a single card or a single die avoided.

7.3 Performance optimization requirements

The performance optimization of the scheduler meets the following requirements: a) it shall be able to collect, analyse and aggregate performance data covering the following devices, including but not limited to the control CPU, the scheduling CPU, the AI accelerator, peripheral devices and memory; b) for multi-card scheduling, it should be able to refer to the attributes and state of the collective communication component during scheduling; c) it should provide a system performance analysis tool for AI tasks on heterogeneous computing devices.

7.4 Scheduling policy requirements

The scheduling policy of the scheduler should take account of the following factors: a) keeping the average utilization of global resources at its highest; b) where the computing resources scheduled involve several AI accelerators, framing the scheduling policy according to the topology of the cards, for example affinity scheduling; c) supporting user-defined scheduling.

7.5 Interface requirements

Interface types. The virtualization interfaces shall include: a) the interfaces between the scheduler and computing resources such as CPU, GPU, FPGA and NPU; b) the interface between the scheduler and the machine learning framework in the AI application.

Interface description — computing instance creation interface. The creation of computing instances shall conform to the interface requirements listed in Table 1. The table gives, for each interface, its number, name, functional semantic description, parameters, whether each parameter is an input or an output, and whether it is optional or mandatory. The interfaces cover setting the computing instance obtained by virtualized division, creating a computing instance to a specified specification on demand, and setting the virtualization mode on the physical machine. The parameters named in the table include the instance representation template, computing capability such as the number of tensor computing cores, memory capacity, the digital vision preprocessing module included at virtualization, the device ID, the chip ID, the number of the virtual device to be created, container mode for a computing instance mounted to a container, and virtual machine mode for a computing instance mounted to a virtual machine.

Computing instance monitoring interface. The monitoring of computing instances shall include: a) obtaining the information of a single physical AI accelerator card, see Table 2, whose output parameters are the maximum number of computing instances available after virtualization, the number of computing instances already carrying out computing tasks, and the resource occupancy, comprising the proportion of resources taken by the computing instances carrying out tasks and the proportion of resources still available for virtualization; b) obtaining the specification of a computing instance, the interface being described in Table 3, whose output parameter is the instance specification; c) obtaining the state of a computing instance, the interface being described in Table 4, whose output parameter is the instance state.

8.1 Technical requirements for operation and maintenance monitoring — AI accelerator card monitoring

AI accelerator card monitoring meets the following. a) It shall be able to monitor the following attributes of the AI accelerator card: computing capability; bandwidth; memory. b) It shall be able to monitor and return the following statistical information on computing instances: the total number of computing instances; the number of available computing instances; the number of occupied computing instances. c) It shall be able to monitor the performance offset of the several computing instances on a single AI accelerator card. Note: where a single AI accelerator card is virtualized into N computing instances of the same specification, N being a positive integer, the reference performance of each instance is one Nth, and the absolute value of the difference between actual performance and reference performance is the performance offset. d) It shall be able to monitor and return the state of the AI accelerator card: the proportion of resources occupied; the proportion available; and whether it is damaged. e) It shall be able to monitor and return the following information about AI tasks: the throughput of the training or inference task; and information on the computing instances occupied by the training or inference task, together with the AI accelerator card on which those computing instances sit.

8.2 Computing instance monitoring

Computing instance monitoring shall meet the following: a) it supports the monitoring of the state of computing instances; b) it supports anomaly detection on computing instances and makes customized detection items open to the user; c) it supports anomaly warning for computing instances and supports user-defined warning thresholds and warning methods; d) it supports the functions of log collection, analysis and visual presentation for computing instances.

8.3 AI task monitoring

AI tasks are monitored by means of a system performance analysis tool. The tool should provide system state prediction or monitoring information, including but not limited to the following.

a) A timeline information graph, presenting the performance information related to the AI task in the form of a time line: computation graph information, that is the execution time of the computation graph, the inference time and the preprocessing time of data such as images; the execution time of operators, that is the start and end time of a single operator and the time it takes; the timeline information of the runtime API for each thread; the timeline information of the running of kernel functions on the AI computing unit; and the timeline information of the running of tasks in task scheduling.

b) A hardware view, presenting the bandwidth information of data transfer, which can help the user or the scheduler find the bottleneck in data processing and transfer; for example, an unduly low read and write bandwidth may indicate a problem in the data movement of the way an operator is written.

c) An AI processor function and CPU function view: collection of hot functions, that is functions executed relatively frequently, on processors of the several forms; and the information presented shall include but not be limited to module name, function name, call stack, number of instructions, proportion of time taken by the number of instructions, and the number of loops of the corresponding sub-item.

d) A summary view, aggregating and presenting the collected information, including but not limited to: the start and end time of the performance data collection and the time it took; basic server performance information, including operating system and CPU information; information on the accelerating device, whether GPU, NPU or FPGA, such as the number of coprocessing CPU cores and the number of AI computing units; memory and high-level cache information, that is the average value of the memory read and write bandwidth over the whole collection process; the utilization of the CPU; system information, including but not limited to total system memory, free space, memory buffer size, cache memory usage, shared memory and virtual memory limits; information on the control CPU, including but not limited to the name of the function running, the module associated with that function, the number of clock cycles for which that function ran, the proportion consumed by the operating system in the virtual environment, the proportion of time in kernel mode processes and the proportion of time in the idle state; information on the AI computing units; operator statistics, including but not limited to the number, names and time taken of operators, the corresponding start and end scheduling times, the memory size of the input and output tensors and the size of the weights; and information related to image and video tasks, including but not limited to the time of the task and the number of frames of the task.

e) The time taken by the data input of the model, by the AI computing task, whether inference or training, and by the data output.

8.4 Log monitoring

Log collection levels. The operation and maintenance monitoring process shall collect logs at the following levels. a) Logs of general level errors are provided: unexpected data or events; errors of wide effect that can nevertheless be handled by the internal module; errors confined within the module; errors that affect other modules but only slightly, such as failure to create a statistics task; and errors that cause a call to fail. b) Logs of warning level are provided, for example a device state that does not agree with expectation but that does not affect the running of the system. c) Logs of normal level are provided, that is information on the normal running of the device. d) Logs of debug level are provided, recording debugging information and assisting development and maintenance personnel in locating problems. e) Logs of key events are provided, for example the start, completion and abnormal termination of a whole-network operation; memory exhaustion; and device temperature exceeding the normal threshold.

Information recorded in a log. The recording of a log shall include the following information: a) the level of the log; b) the name of the module that produced the log; c) the process identifier; d) the process name; e) the time at which the log was printed; f) the log content of each module.

Log processing. The processing of logs shall meet the following requirements: a) it supports the management of accelerating devices, such as addition, deletion and modification; b) it provides the user with log queries, searching the log content by keyword; c) it provides the function of clearing the display; d) it provides the function of downloading and saving logs.

9.1 Test methods — Virtualization tests

Performance isolation test — overview. The performance isolation test is used to measure the performance deviation, that is the absolute value of the difference between actual performance and theoretical performance; the performance indicator is throughput. Example: where a single user or task has a single AI accelerator card to itself, the performance of training or inference is p. Where 25 percent of the computing resources are allocated to a computing instance, its performance should in theory be p divided by four. A virtualization scheme should bring the actual computing capability of the computing instance as close as possible to the theoretical value. The greater the deviation, that is the amount by which it falls below the theoretical value, the poorer the performance isolation. The effect of performance isolation is measured by the deviation of the actual value from the theoretical value. Note: where a single AI accelerator card is divided by virtualization into several computing instances, those instances are allocated to several users. For any AI computing task, each computing instance is allocated on demand, and the performance of the several computing instances on the same accelerator card does not interfere with one another.

Performance isolation test — test method. The performance isolation test shall be carried out according to the following rules. a) Select a particular AI task and carry out the test. b) Measure the baseline performance as the theoretical value. c) For the particular AI task, allocate a certain proportion of the computing resources of a single AI accelerator card. Note: the value of the proportion is fixed at the time of the particular test, but for the virtualization tests of different AI accelerator cards within the same test batch it shall be kept consistent. d) Set the maximum number of tasks n, n being a positive integer, to be executed on the single AI accelerator, have that task use the computing resources already allocated, and allocate the remaining computing resources to the remaining n minus one tasks. e) Calculate the performance deviation in accordance with 9.1.1.1. f) The mode of the performance isolation test is fixed by the permutation and combination of the following factors: training or inference; single-model or multi-model load; and whether fully loaded or not.

Performance isolation test — test loads. The test loads for the performance isolation test comprise the following. a) Baseline performance: a single-model load has a single AI accelerator card to itself, without virtualization, and the baseline performance is measured; the loads are given in Table 5, which lists four test items using the models MobileNet, ResNet50, VGG16 and InceptionV3, each with 100 percent of the resources allocated, that is without virtualization, and each on the ImageNet2012 dataset. b) Single-model fully loaded test: a single AI accelerator card is virtualized in proportion into several computing instances, each computing instance running the same AI task; the loads are given in Table 6, which sets, for each of the four models, the number of loads, the dataset and the allocation ratio between the instances. c) Single-model not fully loaded test: a single AI accelerator card is virtualized in proportion into several computing instances but with room left over, each computing instance running the same model; the loads are given in Table 7, which sets, for each of the four models, the number of loads, the dataset and the proportion of the card allocated to each instance. d) Multi-model test: a single AI accelerator card is virtualized in proportion into several computing instances, room being allowed to remain, each computing instance running a different model; the loads are given in Table 8, which specifies three test items on the ImageNet2012 dataset: MobileNet at 25 percent with ResNet50 at 25 percent and VGG16 at 50 percent; MobileNet at 50 percent with MobileNet at 25 percent and VGG16 at 25 percent; and MobileNet at 25 percent with ResNet50 at 25 percent and InceptionV3 at 50 percent.

Dynamic task scaling test. In actual application scenarios tasks may be added dynamically, that is tasks arrive, or deleted, that is tasks finish executing or are cancelled; the test examines the effect of the dynamic change in the number of tasks on performance. As the number of tasks increases, the overall performance fluctuation, see 9.1.4.4, should be kept as small as possible. The test shall be carried out according to the following rules: a) the scaling test is carried out on identical training or inference tasks, each task being completely the same in model, dataset, training method or inference mode, such as offline model compression and manner of deployment; b) the number of tasks takes the values 1, 2, 4, 8 and 16; c) the throughput at each number of tasks is recorded. The test loads are given in Table 9, which lists four test items using the models MobileNet, ResNet50, VGG16 and InceptionV3, each run at 1, 2, 4, 8 and 16 tasks on the ImageNet2012 dataset.

Performance loss test. For a single AI accelerator card, the performance loss is the difference between the fully loaded performance when it is used without virtualization and the fully loaded performance after virtualization. The test shall be carried out according to the following rules: a) the number of computing instances of the AI accelerator card under test shall be not less than 3; b) the mode of the performance loss test is fixed by the permutation and combination of the following factors: several computing instances with the same load, and several computing instances with different loads. The test loads for the performance loss test are those of items a) to c) of 9.1.1.3.

Test indicators. Throughput: the definition and method of calculation of throughput follow 6.3.3, for training, and 7.3.3, for inference, of GB/T 45087-2024.

Performance deviation: the performance deviation is the absolute value of the difference between the theoretical performance and the actual performance of the AI computing resources obtained after virtualization, calculated by formula (1). Note: where more than one indicator acts at the same time, the test is set up so that the other indicators are the same or unchanged. In the formula, delta p is the performance deviation of a single AI accelerator card after virtualization; n is the number of computing instances; P is the theoretical performance, that is the baseline performance; a sub i is the proportion of the whole card allocated to each computing instance, which may take the values 12.5 percent, 25 percent, 50 percent or 75 percent; and p sub i is the actual performance of each computing instance.

Performance loss: the performance loss is the difference between the fully loaded performance when used without virtualization and the fully loaded performance after virtualization, calculated by formula (2). In the formula, C loss is the performance loss; T sub v is the execution time of the AI task on the computing instance; T sub p is the execution time of the same AI task on the physical machine without virtualization; core sub p is the number of computing units of the physical machine; and core sub v is the number of computing units of the computing instance after virtualization.

Performance fluctuation: the performance fluctuation is the ratio, where a single physical accelerator card is virtualized into several computing instances of the same specification running the same AI task at the same time, of the mean of the longest execution time among all the computing instances to the mean of the shortest, calculated by formula (3). In the formula, the symbol denotes the performance fluctuation ratio after virtualization; n is the nth computing instance; N is the total number of computing instances; and T is the length of time for which the particular computing task is run.

9.2 Scheduling tests

Overview. The scope of the scheduling test covers the following. a) Cluster utilization: the use made of all the AI accelerator cards, including AI accelerator cards already taken offline through failure and those not taken offline but with degraded performance through failure. The closer to 1.0 the better. b) Resource scheduling success rate: the ratio, within a given period, of the number of successful scheduling operations to the total number of resource requests. Note 1: it is a precondition that the actual physical resources match the resources requested, and that the amount of resource requested does not exceed the amount of actual physical resource. c) Task resource utilization: the use made of the computing resources actually allocated to a single AI task. Note 2: the utilization of a GPU generally means the utilization of its stream processors.

Test method. The scheduling test is carried out according to the following rules: a) the test of AI computing cluster scheduling shall specify a definite test period, such as 7 natural days, 15 natural days or 30 natural days; b) in a comparative test, the test load model, the dataset, the optimization method and so on shall be kept consistent; c) the mode of the task load is fixed by the permutation and combination of the following factors: amount of data; amount of computation, divided according to the size of the load model; scenario, divided according to the functional coverage of the model, such as the resnet family or the MobileNet family; and the pattern in which tasks, whether training or inference, arrive, including offline, uniform in time and Poisson arrival; d) the detection period for cluster utilization is one month, and for a particular single AI accelerator card the utilization is detected within that period at a fixed frequency, such as 1 s, and each measurement recorded.

Test loads. The scheduling test uses the following loads. a) Single loads, comprising: vision loads, including image and video tasks; natural language processing loads; and ordinary machine learning loads, which are optional. b) Mixed loads, that is the three kinds of single load combined in a particular proportion, on which the test is carried out.

Test indicators. Cluster utilization is calculated by formula (4). In the formula, Q is the cluster utilization; k is the number of AI accelerator cards in the AI computing cluster; and q sub k is the utilization of a single AI accelerator card.

The utilization of a single AI accelerator card is calculated by formula (5). In the formula, q sub k is the utilization of the particular AI accelerator card within the statistical period; N sub b is the number of times that AI accelerator card was occupied; and q sub u is the instantaneous utilization at the moment of sampling, the utilization when occupied being taken as 1.0 in systems where instantaneous utilization cannot be detected.

Task resource utilization is calculated by formula (6), in which k is the number of AI accelerator cards allocated to the particular task. For a dynamic task, that is one that releases resources or applies for new resources while executing, the measurement is made segment by segment and a weighted average taken by length of time. In the formula, Q task is the task resource utilization; q sub k is the utilization of a single AI accelerator card within a given period of time, during which the number of resources allocated to that task is constant, calculated as in 9.2.4.2; T is the total execution time of that task; j denotes the jth period of time during which that task executes, over which the number of resources allocated to it is constant; k is the number of AI accelerator cards allocated to that task within the jth period; and T sub j is the length of the jth period.

A.1 Reference architecture for NPU virtualization

NPU virtualization includes but is not limited to the following schemes. Note: in the context of this document, Host means the host machine and Guest the guest machine.

Scheme one, software-emulated vNPU virtualization. A virtual NPU device is created through a software framework and, by way of the QEMU device emulation layer, the vNPU device is emulated as a standard PCIE device for the device driver of the Guest OS or for the virtual machine to use, the manner of use being consistent with NPU device pass-through virtualization. The configuration space of the PCIE device is as follows, and the overall device emulation scheme is given in Figure A.1. 1) PCIE configuration emulation: the vendor ID, device ID and so on of the PCIE device are emulated in accordance with the PCIE protocol, presenting to the Guest OS driver a virtual device with the same functions as the physical device. 2) BAR space: the channel through which the Host driver and the Guest driver share data, used to send and receive messages and the like. 3) Interrupt notification mechanism: the Guest sends a message to notify the Host, and the Host notifies the Guest by interrupt. 4) DMA memory access mechanism: the data copy process converts the GPA address into the IOVA address used to initiate DMA access.

Scheme two, software and hardware cooperative SR-IOV virtualization. SR-IOV is a technical standard for hardware-assisted IO virtualization which allows PCIE devices to be shared efficiently between virtual machines; because it is implemented in hardware, IO performance close to that of real hardware can be obtained. The SR-IOV architecture device allows one physical device to support several virtual functions, and it is concerned more with reducing the hardware cost of each additional function. SR-IOV introduces two new function types: physical function and virtual function. A device with SR-IOV capability can, through configuration, present several functions in the configuration space; the number of independent VFs it supports is configurable, and each function has its own configuration space including BARs. Through configuration space matching, the VMM can assign one or several VFs to a VM; the basic principle is shown in Figure A.2.

Scheme three, pass-through virtualization. The VFIO user-mode driver framework is used: the physical NPU device provides device access interfaces and an IOMMU interface to user mode through the VFIO framework, and is able to map the DMA address space into the address space of the virtual machine process, so that the virtual machine can access the device safely. See Figure A.3.

A.2 Reference architecture for CPU virtualization

CPU virtualization is divided into software-based virtualization and hardware-assisted virtualization. Software-based CPU virtualization emulates every instruction in software; two software virtualization techniques are in common use, privilege compression and binary code translation. This purely software way of virtualizing by trapping and then emulating has relatively low performance.

Hardware-assisted CPU virtualization technology introduces the VMX mode, which comprises a root operation mode and a non-root operation mode. Both modes support different running levels, such as Ring0, Ring1 and Ring3. The benefit of the VMX mode is that the Guest OS runs at Ring 0, which means that its core instructions can be issued directly to the hardware layer for execution, while the execution of privileged and other sensitive instructions is assisted by the hardware and switched directly to the VMM, so that performance is considerably improved.

On a virtualized platform, the several virtual CPUs, below called vCPUs, used by a virtual machine, the Guest VM, may share the same physical CPU, below called the pCPU. The VMM is responsible for scheduling the vCPUs; once a vCPU has been scheduled and obtained the use of a pCPU, the Guest OS running on that vCPU can in turn schedule the several threads and processes within the OS. That is, the several threads and processes in the Guest OS time-share the vCPU, and the several vCPUs in turn time-share the pCPU.

Figure A.4 gives the reference architecture for CPU virtualization, divided into the hardware layer, the physical host layer and the virtual machine layer. The virtual machine layer schedules task threads on the vCPU and the physical host schedules CPU threads on the hardware CPU; the framework as a whole has two levels of CPU scheduling, the vCPU in the Guest OS being responsible for scheduling task threads and the Host VMM being responsible for the other level, namely the scheduling of the vCPU on the physical CPU.

For a physical CPU, one CPU has several cores and one core generally corresponds to one thread, but with hyper-threading technology one core can correspond to two threads, that is it can run two threads at the same time and process two tasks in parallel. The number of vCPUs may exceed the number of physical CPUs; the virtual machine can make full use of the CPU resources of the Host for the corresponding business processing, and operation and maintenance personnel can accordingly control the use of CPU resources so as to achieve flexible scheduling.