GB/T 47476-2026Intelligent computing - Test methods for graph computing performance (English PDF)

5.2.1 Graph Query System

a) Throughput Throughput represents the number of query requests that a graph query system can process per unit of time, and is used to measure the overall throughput of a graph query system. Processing capacity. Calculated according to formula (1).

5.2.2 Graph Analysis System The performance test of the graph analysis system includes the following metrics.

a) Algorithm running time Algorithm runtime represents the time consumed by a graph analysis system to execute a specified graph analysis algorithm on a selected test dataset and obtain results. The time interval is used to measure the efficiency of the graph analysis system in handling graph analysis tasks. It is calculated according to formula (5).

5.2.3 Graph Learning System The performance test of a graph learning system includes the following metrics.

a) Model effect Model performance indicates the quality of the model obtained after the graph learning system completes the graph learning training task. Depending on the task type, the model performance varies. The performance indicators of the different models vary. 1) For classification tasks, the model performance metrics include precision, recall, and F1 score, which are calculated according to formulas (6) to (8), respectively.

6.1.1 Test Environment

6.1.1.1 Test Environment Components The test environment should include the system under test, the test-driven system, the test network, and test auxiliary tools, the components of which are as follows:

a) The system under test. the graph query system, graph analysis system, or graph learning system to be tested;

b) Test-Driven System. This includes the test load (data), the test driver that loads the test load to perform performance testing on the system under test, and... The software and hardware environment required for its operation;

c) Test Network. Consists of network devices and their configurations used to build the test network environment;

d) Test support tools. including various tools that support the testing process, such as data import, monitoring and logging, and data analysis.

6.1.1.2 Comparability of Test Environments The hardware and software environments of different systems under test should be kept consistent to ensure the comparability of test results. When the consistency of the hardware environment of the system under test cannot be guaranteed, the computational costs of the system under test should be disclosed to facilitate the evaluation of its performance-price ratio. Node quantity information and compute node hardware and software configuration information, wherein the compute node hardware and software configuration information of the tested system should specifically include the following. aspect.

a) Software configuration information. 1) Operating system type and version information; 2) JDK version information (if applicable); 3) MySQL version information (if applicable).

b) Hardware configuration information. 1) CPU information, including model, number of cores, and clock speed; 2) Memory information, including capacity and type; 3) Hard drive information, including type, capacity, and interface type; 4) Network interface card (NIC) information, including model and bandwidth; 5) Switch information, including model and switching capacity; 6) GPU information (if applicable), including model, number of cores, computing power, and interface type; 7) Video memory information (if applicable), including capacity, type, and bandwidth; 8) Information on dedicated chips (if applicable), including model and main performance parameters.

6.1.2 Test Prerequisites Before conducting formal performance testing, the functionality of the system under test should be verified.

6.2 Graph Query

6.2.1 Test Code The system under test should preferably use a query language that can flexibly express query objectives. The code written for graph query performance testing should be disclosed, including... This includes, but is not limited to, query languages, SDKs, and drivers.

6.2.2 Test Dataset The test dataset used for performance testing should be divided into two parts. One part should be loaded into the graph query system in batches before performance testing to serve as the base dataset. The basic data, which should comprise at least 80% of the total dataset, and the remaining portion, used as test load data during performance testing, constitute the other part. The dataset should not exceed 20% of the total dataset. The test dataset should also meet the following requirements.

a) The test dataset is constructed and generated based on a representative real-world graph application data model. The dataset simulates real-world application scenarios. The data status during typical time periods in the scene reflects the characteristics of real-world scene data, including data volume, percentage of various types of vertices, etc. Attribute values, relationship distribution, and typical query load patterns, etc.

b) The test dataset is scalable, capable of generating test datasets of different sizes based on different SF values to adapt to different requirements. A graph query system for models.

6.2.3 Test-Driven Test-driven testing is used for process control at various stages of testing, such as result verification and performance testing, and should meet the following requirements.

a) It includes modules for data loading, load generation, result statistics, result verification, and report generation;

b) It has functions such as result correctness verification and performance stress testing;

c) It has parameter configuration functions, which can be used to adjust the test load pressure, load mixing ratio, and test duration. Adjustment;

Note. Load mixing ratio refers to the ratio of different types of operations in the test load, such as the ratio of read and write operations, or the ratio of operations with different query complexities.

d) Supports loading test load data as input for graph queries;

e) Supports various query runtime statistics during performance stress testing and provides an automatic test report generation function;

f) Provides a good integration interface to facilitate quick access by the system under test.

6.2.4 Test Load Test loads used for performance testing should include various query operations targeting points, edges, attributes, and matching rules. Complex queries within the load are also important. Simple queries, as well as the ratio, order, and time interval of read and write operations, should simulate real-world application scenarios.

6.3 Graphical Analysis

6.3.1 Test Code The system under test should provide a test-specific interface or executable program that can be invoked through the test driver. The code written for test driver invocation... The code should be disclosed.

6.3.2 Test Dataset The test dataset serves as the foundational data for loading into the graph analysis system for algorithm testing and meets the following requirements.

a) The test dataset should use representative real-world or generated graph data, and should reflect the characteristics of real-world scene data;

b) The test dataset should include graph topology data and preferably include the correct execution results of the test algorithm to verify the graph algorithm of the system under test. Is the result of the operation correct?

6.3.3 Test Algorithm The test algorithm is a benchmark used to test the functionality of the basic operator model contained in the system under test, and should meet the following requirements.

a) Supports graph traversal algorithms;

b) Supports path lookup algorithms;

c) Supports importance analysis algorithms;

d) Supports community detection algorithms;

e) Supports correlation analysis algorithms;

f) Supports graph matching algorithms.

6.3.4 Test-Driven Test-driven programming is used for process control in stages such as data preprocessing, loading, algorithm execution, result output, and result verification. It should meet the following requirements. Require.

a) It has modules for data preprocessing, loading, algorithm execution, result output, and result verification;

b) Supports custom definitions of intermediate file formats during preprocessing and loading stages;

6.4 Graph Learning

6.4.1 Test Code Since there is currently no unified programming language for graph learning models, the system under test can use any language capable of executing the functions of the graph learning model. For APIs, the code written for performance testing should be disclosed.

6.4.2 Test Dataset The test dataset used for performance testing should be divided into two parts. one part should be able to be batch-loaded into the graph learning system before performance testing. This portion serves as the foundational data, comprising at least 80% of the total dataset; the other portion is used as test load data during performance testing. A portion of the dataset should not exceed 20% of the total dataset. The test dataset must simultaneously meet the following requirements.

a) The test dataset should be constructed and generated based on a representative real-world graph application data model. The dataset should simulate real-world applications. The dynamically evolving graph structure used in the scene should be able to reflect the characteristics of real-world scene data, including the proportion of various vertices, attribute values, and relationships. Distribution and its temporal trends, including annotation information or mechanisms for generating labels;

b) The test dataset should include graph topology data and corresponding feature data, and preferably include correct inference results corresponding to downstream test tasks. Data is used to verify the correctness of the inference results of the graph learning model trained by the tested system.

6.4.3 Test Model The test model library is a benchmark used to test the integrated functions contained in the system under test that perform computational tasks, and should meet the following requirements.

a) Supports convolutional graph neural networks;

b) Supports recurrent graph neural networks;

c) Support graph attention networks;

d) Supports graph residual networks;

e) Supports hypergraph neural networks;

f) Supports heterogeneous graph neural networks.

6.4.4 Downstream Tasks The downstream task library serves as a benchmark for evaluating the performance of the system under test in solving real-world problems and should meet the following requirements.

a) Supports categorized tasks; Note

1.Classification refers to dividing the vertices/edges/graph itself in the graph data into a specified or unspecified number of disjoint groups.

7.1.1 Response timeliness and throughput

7.1.1.1 Test Objective This test is used to determine the timeliness of query operations and the system throughput of the graph query system.

7.1.1.2 Test Procedure The test steps are as follows:

a) Select the size of the test dataset.

b) Import the basic data portion of the test dataset into the system under test. Assuming the imported data is complete, the data import process should be recorded. Long import.

c) Select the test load control parameters and set the response time tset_response.

d) Use test-driven testing to test the system under test. Testing should last at least 2 hours to obtain stable and reproducible test results. Before starting, the graph query system should be reset and cleaned up to ensure the system environment under test is in its initial state. Testing Phase The test is divided into two phases. data warm-up and formal testing. The query statements used in the warm-up phase should be the same as those used in the formal testing phase. The query statements should be independent and should not be duplicated or equivalent. During the data warm-up phase after the test begins, the data warm-up duration should be recorded. During the formal testing phase, twarm_up should record the request time (tquery_i_start) and execution end time for each query operation. tquery_i_end.

e) After the test is completed, calculate the time consumption of each query operation in the formal test phase. tquery_i = tquery_i_end - tquery_i_start, and statistically analyze the query time. The query operation takes longer than the number of queries for which the set response time tset_response is set.

f) Calculate response timeliness and throughput.

7.1.1.3 Test Report The test report should include the following information.

a) Test result information, including. 1) Query response timeliness; 2) Throughput of the system under test.

b) Configuration information of the system under test, including. 1) Information on the number of computing nodes in the system under test and its single-machine or distributed deployment architecture; 2) Software configuration information of the computing nodes of the system under test, see 6.1.1.2a). 3) Hardware configuration information of the computing nodes of the system under test, see 6.1.1.2b). 4) The graphical model used in the system under test.

c) Test task information, including. 1) Test dataset information, including dataset name, number of vertices, number of edges, source, and domain; 2) Test load information, including dataset size and load mixing ratio; 3) The query language used in the test.

d) Test process information, including. 1) Data import time; 2) Data warm-up time; 3) Query execution time; 4) Query the number of operations; 5) Query response time distribution, including average response time, median response time (50%), 90% response time, and 95% response time. Time, 99% response time, and maximum response time.

e) Status information of the system under test, including. 1) Processor operating status, including average and maximum CPU utilization and average GPU utilization during test execution. and maximum utilization rate (if applicable), average utilization rate and maximum utilization rate of dedicated chips (if applicable); 2) Storage usage, including memory, video memory (if applicable), and external storage usage before, during, and after data import, as well as the usage during test execution. Memory, video memory (if applicable), and external storage usage during and after the test; 3) I/O conditions, including average and maximum disk I/O and average and maximum network I/O during the test execution process.

7.1.2 Storage Usage

7.1.2.1 Test Objective This test is used to determine the storage usage of the graph query system under maximum throughput conditions, including memory, video memory (if applicable), and external memory. Storage occupancy status.

7.1.2.2 Test Procedure This item does not need to be tested separately; relevant records should be made when performing test 7.1.1.

7.1.2.3 Test Report The test report should include at least the following information.

a) Test result information, including. 1) Instantaneous storage usage values of the tested system's memory, video memory (if applicable), and external storage during the first 10 minutes of stable operation after startup; 2) The maximum storage usage of memory, video memory (if applicable), and external storage during the data import process of the system under test; 3) Instantaneous storage usage of memory, video memory (if applicable), and external storage 10 minutes after the data import of the tested system is completed; 4) The maximum storage usage of the tested system in memory, video memory (if applicable), and external storage during the test; 5) Instantaneous storage usage values of memory, video memory (if applicable), and external storage of the system under test 10 minutes after the test ends.

b) Storage configuration information of the system under test, including the storage capacity configuration of memory, video memory (if applicable), and external storage.

7.1.3 Extensibility

7.1.3.1 Test Objective This test evaluates the ability of a graph query system to improve performance by increasing the number of compute nodes under a fixed query load, and its ability to improve performance during query operations. The ability to maintain stable performance when the query load increases proportionally with the number of compute nodes.

7.1.3.2 Test Procedure The test steps are as follows:

a) Select a fixed-size test dataset and baseline query load.

b) Determine the number of computing nodes in the test sequence (preferably including 1, 2, or 4 computing nodes).

c) Perform the tests according to the following configurations. 1) Under a fixed baseline query load, run tests with different numbers of compute nodes and record the execution time of each test. Tquery_scal_i; 2) Under the condition that the query load increases proportionally with the number of compute nodes, test different numbers of compute nodes and corresponding query loads. Perform load testing and record the execution time of each test, Tquery_scal_j.

d) Based on the test results of each configuration group, the performance speedup when the number of compute nodes increases, and the query load versus the number of compute nodes. The performance retention factor when the increase is proportional.

7.1.3.3 Test Report The test report should include at least the following information.

a) Test result information, including. 1) Performance speedup as the number of compute nodes increases; 2) Query the performance retention coefficient when the load increases proportionally with the number of compute nodes.

b) Configuration information of the system under test under each test configuration. The configuration information of the system under test shall include the contents specified in 7.1.1.3b).

c) Test task information under each test configuration. The test task information should include the content specified in 7.1.1.3c).

d) Test process information under each test configuration. The test process information should include the content specified in 7.1.1.3d).

7.2.1 Algorithm Execution Time

7.2.1.1 Test Objective This test is used to determine the time taken for a graph analysis system to execute a specified graph analysis algorithm on a selected test dataset and obtain results.

7.2.1.2 Test Procedure The test steps are as follows:

a) Perform data preprocessing on the selected test dataset, process the intermediate files required by the graph analysis system, and record the data preprocessing stages. Segment execution time Tpre_processing;

b) Select a test algorithm, specify fixed algorithm configuration parameters, and use a test driver to load, execute, and process the algorithm on the system under test. Operations such as output are recorded, including the data loading time Ttest_data_load, algorithm execution time Texec, and result output time of the tested system. Interval Toutput;

c) Verify the output of process

b) and record the verification results;

d) For the same test algorithm, execute processes

c) 3 to 5 times, calculate and record the average data loading time Tavg_pre_processing, and average... Average algorithm execution time Tavg_exec, average result output time Tavg_output;

e) For multiple test algorithms on the same test dataset, execute steps b), c), and

d) sequentially.

7.2.1.3 Test Report The test report should include at least the following information.

a) Test result data, i.e., the average algorithm execution time under a given test dataset and test algorithm.

b) The configuration information of the system under test shall include the contents specified in 7.1.1.3b).

c) Test task information, including. 1) Test dataset information, including dataset name, number of vertices, number of edges, source, and domain; 2) Parameter configuration for testing algorithm execution; 3) The test language used in the test.

7.2.2 Storage Usage

7.2.2.1 Test Objective This test is used to determine the storage usage of the graph analysis system during the execution of graph analysis algorithms, including memory, video memory (if applicable), and... External storage usage.

7.2.2.2 Test Procedure This item does not need to be tested separately; relevant records should be made when performing test 7.2.1.

7.2.2.3 Test Report The test report should include at least the following information.

a) Test result information, including. 1) Instantaneous storage usage values of the tested system's main memory, video memory (if applicable), and secondary storage after 10 minutes of stable operation; 2) The maximum storage usage of the tested system at each stage of data preprocessing, loading, algorithm execution, and result output; 3) Instantaneous storage usage values of memory, video memory (if applicable), and external storage of the system under test 10 minutes after the test ends.

b) Storage configuration information of the system under test, including the storage capacity configuration of memory, video memory (if applicable), and external storage.

7.2.3 Extensibility

7.2.3.1 Test Objective This test evaluates the ability of a graph analysis system to improve performance by increasing the number of computing nodes while maintaining a fixed problem size. The ability to maintain stable performance as the problem size grows proportionally with the number of computing nodes.