GB/T 42382.2-2026Information technology - Neural network representation and model compression - Part 2: Large scale pre-trained models (English PDF)

6.1 Syntax Description

6.1.1 Principles This document defines the syntax for representing large-scale pre-trained models, from coarse-grained to fine-grained, that is, from model structure definition, computation graph definition, to... Node definitions, nested layer by layer, construct the basic grammatical description of the entire large-scale pre-trained model. This representation grammar should be defined by a specific computing system. This is completed within a deep learning platform and related hardware and software, following these principles.

a) When implementing the computing system, it is necessary to adjust the syntactic elements according to actual needs, including but not limited to. 1) Keyword (parameter) naming; 2) Operator naming; 3) Data type.

b) The definition of necessary hierarchical levels needs to be considered, including. Model structure definition; 2) Definition of computational graph; 3) Basic data type definitions

6.1.2 Model Structure Definition The model structure represents the basic information and network architecture of the neural network model. The technical parameters describing the model structure are shown in Table 1.

6.1.4 Operation Node Definition The operation node definitions are shown in Table

3.The operation field defines the name of the operation node; input and output are the output parameters of the operation node. Input variable nodes and output variable nodes; attribute is the attribute of the operation node.

6.2.1 Representation of Operations

6.2.1.1 General Rules 6.2.1.1.1 Principles for Using Calculation Operations The computational operations contained in large-scale pre-trained models are implemented by specific computing systems and should follow the following usage principles.

a) Make adjustments as needed, including but not limited to. 1) Keyword (parameter) naming; 2) Operator naming; 3) Data type support range

b) Consider the elements involved in the operation, including. 1) Supported parameters; 2) Supported types. 6.2.1.1.2 Classification of Operations Operators for large-scale pre-trained models can be divided into the following three categories.

a) General operators for large-scale pre-trained model networking, including but not limited to embedding, layer normalization, linearity, and attention. Operators, whose definitions are shown in Tables 4 to 7.

b) General operators for neural network models, including but not limited to basic mathematical operators such as reshape and concat, and neural network operators. For detailed definitions of ReLU and Softmax, please refer to other relevant operator standards; they will not be repeated in this document. See also [link to relevant documentation]. GB/T 42382.1-2023

c) Communication operators for training large-scale pre-trained models, including but not limited to reduce, allreduce, reduce_scatter, etc. The operators allgather, broadcast, send, and recv are defined as shown in Tables 8 to 14.

6.2.1.2 Definition of Basic Operations This section defines common computational operations used in large-scale pre-trained models. The specific definitions are as follows:

6.2.3 Multimodal Operation Representation

6.2.3.1 Overview 6.2.3.1.1 Core Components Multimodal large-scale pre-trained models are large models that integrate data from multiple modalities for understanding and generation. Their core components are... It includes an input layer, a modality-specific encoder, a modality fusion module, a modality alignment module, a decoding layer, and an output layer. 6.2.3.1.2 Input Layer Different modal inputs correspond to unique input processing modules. The data input processing module includes.

a) Text data. word segmentation and embedding processing;

b) Image data. Pixel value normalization and image enhancement.

c) Audio data. Convert to a spectrogram or other time-frequency representation. 6.2.3.1.3 Modality-Specific Encoder Modality-specific encoders include, but are not limited to.

a) Text encoder. Typically consists of a word embedding layer, a position embedding layer, and multiple Transformer encoders, capable of converting text data... Transformed into high-dimensional feature representations, including BERT, GPT, RoBERTa, etc.;

b) Image encoder. Typically contains convolutional layers (or self-attention layers) and pooling layers, capable of extracting multi-level features of the image, including... ResNet, Vision Transformer (ViT), etc.;

c) Audio encoder. Typically processes audio signals using convolutional layers or Transformer structures to extract meaningful audio features. Including Wav2Vec, Mel-Spectrogram, MFCC, etc.;

d) Video encoder. Typically, it extracts rich video features by processing temporal and spatial information in video data, including... C3D, I3D, etc. 6.2.3.1.4 Modal Fusion Module Multimodal data fusion strategies include.

a) Early fusion. Fusion is performed at the input layer or coding layer. Common methods include concatenation and weighted averaging.

b) Intermediate Fusion. Fusion is performed in the intermediate layers of the model, typically using a multi-layer cross-attention mechanism at different... Information exchange between modalities;

c) Late-stage fusion. Fusion is performed at higher levels of the model or at the output layer, typically using a separate decision module to process the data for each modality. Process the data and then merge the results. 6.2.3.1.5 Modal Alignment Module Modality alignment refers to mapping data from different modalities to the same feature space. Alignment methods include, but are not limited to.

a) Co-occurrence analysis. Aligning data from different modalities using co-occurrence matrices or co-occurrence graphs.

b) Contrastive learning. By constructing positive and negative sample pairs, the similarity between features of the same modality and different modalities is maximized;

c) Alignment transformation. Using linear or nonlinear transformations, features of different modalities are aligned to the same representation space. 6.2.3.1.6 Decoder and Output Layer Decoders and output layers include, but are not limited to.

7.2 Optimization of Large-Scale Pre-trained Model Structure

7.2.1 Nested Structure Nested structures utilize an outer Transformer and an inner Transformer to extract global and local features, respectively. The approach involves using an outer Transformer module to model the relationships between image patches, and an inner Transformer module to model the relationships between them. A technical solution for constructing rich visual representations is achieved by modeling the relationships between sub-image patches. The nested structure preserves the image patch layer... It extracts information at both the surface and pixel levels, significantly improving the model's ability to model local structures, thereby enhancing the model's overall performance. The recognition effect of the type.

7.3.1 Quantification

7.3.1.1 Overview Quantization of large-scale pre-trained models is a key method to improve computational efficiency and reduce model complexity. The core of quantization lies in reducing the model size... The precision of parameter representation in the model is reduced by decreasing numerical precision to decrease storage requirements and accelerate the computation process, while maintaining model performance as much as possible. Quantization techniques can be combined with various model optimization strategies, such as pruning, which can prune the model before quantization to further optimize it. To improve model efficiency, after quantization, each weight of the model is converted to a lower-precision format, for example, using only a smaller number of bits. Each weight value is represented by a quantization method. Especially for models that have already been simplified through pruning, quantization can be directly applied to these preserved weights. The remaining weights are used to achieve additional efficiency improvements. During the quantization process, the degree of quantization can be determined based on the importance of each weight. This is similar to the strategy of selecting retention weights during pruning.

7.3.1.2 Quantification Criteria 7.3.1.2.1 Criteria for Affine Transform Quantization Algorithm of Large-Scale Pre-trained Models Affine transformation quantization algorithms can be used to optimize training-free quantization of large-scale pre-trained models by directly applying equivalent affine transformations. The model is optimized; at the same time, the scope of optimization is expanded to include transformations such as weight scaling and translation, thereby significantly reducing errors in the quantization process. This algorithm utilizes the inverse matrix of the affine transformation to maintain the equivalence of the model output before and after quantization, and introduces a method based on the Levy-Desplanques theorem. A progressive mask optimization method is used to ensure the stability of the matrix optimization process. The specific algorithm flow is as follows:

8.2 Model Encapsulation Representation

8.2.1 Model Encapsulation Representation Structure Diagram The model encapsulation representation structure diagram is shown in Figure 2.

8.2.6 Encryption and Decryption of Model Encapsulation Representation Model encapsulation means that encryption can effectively prevent unauthorized third parties from obtaining, attacking, or tampering with the model. The encryption and decryption of the model encapsulation representation should comply with the content of Chapter

11 Model Protection in GB/T 42382.1-2023.

8.2.7 Incremental Model Update Incremental model update (residual update) is a process that requires updating the model between the edge and the cloud. The new model is generated in the cloud, and then... The model is then distributed to the edge for model propagation to facilitate better edge deployment. Further propagation is required after generating the deep learning model at the edge. Input and fusion models. For multi-model scenarios involving the same or similar tasks, these scenarios typically contain multiple levels of identical/similar models. Similar tasks, these task models typically share the same architecture. For cases where the cloud and edge share parts of the architecture, only the common parts need to be addressed. The weight increments in the architecture are used for residual updates. Other parts need to be transmitted independently. For these similar models, the weights between different models... There is a high correlation. Compared to the original weights, the residuals of the weights are more concentrated and less diverse, making them easier to process with low bit counts. Therefore, model residual update techniques can be used to eliminate information redundancy between information transmissions. The update and transfer process includes the following six steps.

a) Train the base model weights A and transfer the model from the cloud to the edge;

b) Retrain and fine-tune the base model weights A according to actual needs to obtain the target model weights B;