YY/T 1833.2-2022Artificial intelligence medical device—Quality requirements and evaluation - Part 2: General requirements for datasets (English PDF)

3.1 Inspection by attributes

An inspection that is carried out with respect to a specified requirement or group of requirements, or that only classifies unit products as qualified or unqualified, or that only counts the number of unqualified units.

4.1.2.1 Compliance statement

The dataset description shall provide a compliance statement of the data source.

4.1.2.2 Privacy protection

The dataset description shall describe the technical means used to protect the privacy of subjects, such as data de-identification, data anonymization, etc. When appropriate, the dataset description document shall describe the rules for data de-identification or data anonymization.

4.1.2.3 Diversity

The dataset description shall provide a description of the diversity of data sources, such as population, collection location, collection equipment, parameter settings, operator qualifications, collection process, collection time, etc.

4.1.2.4 Principles of compliance for data collection

The dataset description shall provide the regulations, technical standards, clinical specifications, expert consensus or other references on which the data were collected.

4.1.2.5 Data screening

The dataset description shall describe the data entry and exclusion criteria, as well as the methods for data screening, such as manual cleaning and automatic cleaning.

NOTE. See Annex B for examples.

4.1.3 Data preprocessing

When appropriate, the dataset description shall describe the steps and content of data preprocessing.

4.1.4 Dataset annotation

4.1.4.1 Principles for dataset annotation

If the dataset has annotation information, the dataset description shall describe the regulations, technical standards, clinical specifications, expert consensus or other references on which the dataset is annotated.

4.1.4.2 Reference standards

If the dataset has annotation information, the dataset description shall describe the establishment rules, scope, storage format and data specifications of the dataset reference standard. If the reference standard is verifiable, the verification method of the reference standard shall be described.

4.1.4.3 Annotation process

If the dataset has annotation information, the dataset description shall describe the data annotation and quality control process and clarify the decision-making mechanism. In the case of multiple annotations or multiple annotations, the arbitration mechanism for annotation differences shall be described.

4.1.4.4 Other annotation information

If the dataset has annotation information, the dataset description shall describe the scope, data specifications and storage format of other annotation information in addition to the reference standard.

4.1.5 Dataset storage information

The dataset description shall describe the data storage information, such as the dataset storage method and storage path, security control, backup, and recovery instructions. If the dataset is stored using cloud services, the cloud service provider name and qualifications, access path, and usage permission instructions shall be provided.

4.1.6 Dataset user access

4.1.6.1 Access control

The dataset description shall describe the user access control mechanism, such as user type, permission allocation, and authorization mechanism.

4.1.6.2 Access conditions

The dataset description shall describe the conditions required to access the dataset, such as software and hardware configuration, access method, data interface, protocol, tools, etc.

4.1.6.3 Visualization

The dataset description shall describe the visual presentation of the dataset information.

4.1.7 Development management

The dataset description shall describe the governing standards to which the dataset was developed.

4.2 Dataset identification

4.2.1 Identification

The dataset shall display a unique identifier, including the dataset name, version number, and information about the dataset manufacture responsible organization. This can be provided in the form of an attached file and described in detail in the dataset description The dataset description shall state the extent to which the dataset is authentic and trustworthy, including the acquisition and processing of data and metadata. Verifiable evidence shall be presented in written form.

4.3.5 Timeliness

The dataset description shall state the extent to which the timeframes required for each step in the dataset development phase meet expectations, taking into account preprocessing, cleaning, labeling, etc. Provide verifiable metrics in written form.

4.3.6 Accessibility

The dataset description shall state the extent to which the dataset is accessible. Demonstrate verifiable evidence in written form.

4.3.7 Compliance

The dataset description shall state the standard specifications, expert consensus, operating procedures or other references to which the dataset complies.

4.3.8 Confidentiality

The dataset description shall describe the measures related to information security and data confidentiality. Verifiable evidence shall be presented in written form.

The dataset description shall state the resource consumption required to perform the dataset-related tasks. Verifiable compliance evidence shall be presented in written form, such as the software, hardware, and network configuration required for tasks such as accessing, reading data, previewing, and retrieving.

4.3.10 Precision

The dataset description shall describe the closeness of the quantitative information of the data to the true value, taking into account the data elements, metadata, and data annotation results. Provide verifiable indicators in written form, such as spatial/temporal resolution, significant figures, and minimum measurement units.

4.3.11 Traceability

The dataset description shall describe the extent to which the data can be traced, taking into account the data collection history, data annotation history, data access traces, and data change traces.

4.3.12 Understandability

The dataset description shall use terms that are understandable to the users of the dataset.

Provide explanations of the meaning of the data elements, metadata, and annotation results. Present verifiable evidence in written form.

4.3.13 Availability

The dataset description shall state the extent to which the dataset can be used and retrieved by authorized users. Verifiable evidence shall be presented in written form.

4.3.14 Portability

The dataset description shall state the ability of the dataset to be installed, replaced, or moved from one system to another. Maintain the attributes of the existing quality.

Consider the efficiency of data installation, replacement, and movement. Show verifiable evidence in written form.

4.3.15 Recoverability

The dataset description shall state the extent to which the dataset can be recovered.

Verifiable evidence shall be presented in written form. The dataset description may provide measures for data recovery. The dataset description may provide measures to prevent interruption or failure in the use of the dataset.

4.3.16 Representativeness

The dataset description shall analyze the sample composition, proportion, population distribution characteristics, data diversity, and the degree of closeness to the application scenario. Verifiable indicators shall be provided in written form.

5.1 Overview

The content of this document focuses on the quality characteristics and overall risk of the dataset. It is advisable to conduct a quality assessment of the dataset based on its intended use and application scenarios, and form a technical report as a verification of the dataset quality.

5.2 Quality characteristics

5.2.1 Completeness

5.2.1.1 Accuracy

The dataset shall comply with the accuracy statements in the dataset description, such as.

a) Accuracy of recorded information;

b) Accurate, clear and unambiguous text description;

d) Original records, intermediate records and final records shall be consistent.

External consistency refers to the correlation between data from different sources, such as.

a) Data from different sources shall be consistent in terms of data characteristics;

b) Outliers shall be explainable;

c) Data from different sources shall comply with the same regulations, technical standards, medical specifications, and other literature requirements during collection and annotation.

5.2.4 Authenticity

The dataset shall comply with the dataset description with statements about authenticity, such as.

a) Data shall come from real clinical data collection processes. When appropriate, the equipment, personnel, and methods involved in data collection shall comply with technical standards, clinical norms, or expert consensus;

b) Data amplification, data synthesis activities, and results shall be traceable and interpretable;

c) Metadata shall describe the data truthfully.

5.2.5 Timeliness

The time limit for data collection, labeling, circulation, archiving, and change activities shall comply with the statement of timeliness in the dataset description. Dynamically updated datasets shall specify the data update cycle, update method, and update ratio.

If the data involves the time series process in clinical diagnosis and treatment, the rationality of the data in terms of clinical timeliness shall be demonstrated.

5.2.6 Accessibility

Datasets shall meet the access needs within the scope of the intended use and application scenarios of the dataset.

5.2.7 Compliance

Datasets shall comply with the statements about compliance in the dataset description.

5.2.8 Confidentiality

Datasets shall comply with the confidentiality statements in the dataset description.

Take steps to ensure that they can only be accessed by authorized users.

The isolated datasets shall have dataset authorization access mechanism and isolation protection mechanism. There shall be measures to prevent data leakage, data tampering, and data loss, such as data anonymization, physical isolation, data auditing, etc.

5.2.9 Resource utilization

The processing and use of the dataset shall be in accordance with the statement of resource availability in the dataset description.

5.2.10 Precision

The dataset shall comply with the precision statement in the dataset description document.

5.2.11 Traceability

The dataset shall comply with the dataset description regarding traceability, with relevant records, such as.

a) Original data source, metadata source, compliance proof;

b) Data collection activity records;

c) Personnel management records;

d) Data annotation process records;

e) Blind management records;

f) Data circulation records;

g) Data questioning, auditing, deactivation, and correction records;

h) Records of labeling tools and platform usage;

i) Statistical information query of dataset labeling results, including labeling progress, label statistics, labeler progress statistics, difficult case set, etc.;

j) Data service anomaly and failure records;

k) Data maintenance and backup records;

l) Data update records;

m) Cloud service provider name, contact information, cloud service type, etc.

5.2.12 Understandability

Datasets shall conform to the dataset description's statements regarding understandability.

5.2.13 Availability

Datasets shall comply with the availability statements in the dataset description document.

5.2.14 Portability

The dataset shall comply with the statement about portability in the dataset description.

If the dataset is allowed to be used on different platforms and systems, the data quality shall not change with the platform and system.

5.2.15 Recoverability

Measures used to maintain the quality of the dataset and protect against failure events shall be consistent with the statements about recoverability in the dataset description.

5.2.16 Representativeness

The data feature hierarchy, epidemiological statistics, sample source diversity, data diversity, etc. of the dataset shall comply with the representative statements in the dataset description.

5.3 Dataset risk analysis

5.3.1 Selection bias

The dataset manufacture responsible organization shall analyze the representativeness of the dataset (whether the subject population, collection site, equipment selection, parameter settings, etc. represent clinical reality).

5.3.2 Coverage bias

The dataset manufacture responsible organization shall analyze whether the disease composition of the dataset can cover all situations of the target population in the application scenario (single, multiple, concurrent, complex pathology, etc.).

5.3.3 Reference standard bias

The dataset manufacture responsible organization shall verify the data annotation results or the process of establishing the reference standard. Analyze the bias of the reference standard.

5.3.4 Verification bias

The dataset manufacture responsible organization shall describe the verification method of the reference standard. Analyze the correlation, differences and impact between the data annotation process and the clinical diagnosis process.

6.3.6 Timeliness

The extraction and inspection of time information from the sampled sample set shall comply with the requirements of 5.2.5.

6.3.7 Accessibility

Write test cases based on statements about accessibility in the dataset description. Actual operational verification shall comply with the requirements of 5.2.6.

6.3.8 Compliance

Write test cases. Check the compliance of the sampled sample set with relevant regulations, technical standards, and technical specifications, which shall comply with the requirements of 5.2.7.

6.3.9 Confidentiality

Write test cases. The authorization access mechanism and data isolation protection mechanism of the sampled sample set shall comply with the requirements of 5.2.8.

6.3.10 Resource utilization

Write test cases. Under the operating environment specified by the dataset manufacture responsible organization, operational verification of the dataset shall comply with the requirements of 5.2.9.

6.3.11 Precision

If applicable, perform process validation on the dataset annotation tool. The data, metadata, quantitative features included in the annotated results, and the accuracy of the quantitative description of the dataset in the sample set shall meet the requirements of 5.2.10.

6.3.12 Traceability

The traceable records of the sampled sample set shall be checked in accordance with the requirements of 5.2.11.

6.3.13 Understandability

Refer to the dataset description for actual operation. Preview and interpretation of data shall comply with the requirements of 5.2.12.

6.3.14 Accessibility

Refer to the dataset description for actual operation. The use and retrieval of data shall comply with the requirements of 5.2.13.