GB/T 38665.2-2020Information technology - Gesture interaction system - Part 2: System external interface (English PDF)

3.1 Gesture recognition algorithm

A computer program that is used in the process of converting a user's gesture signal into a gesture recognition result.

3.2 Discrete gesture recognition algorithm

A class of gesture recognition algorithm that the recognition module recognizes a piece of gesture data at a time, and obtains the corresponding recognition result.

EXAMPLE: For a gesture interaction system equipped with a handle, the user presses the handle button to start a gesture, and releases the button to end the gesture. The recognition

Number: 2-2

Description: onGestureStarted(void): void

Function: This interface is used to indicate a gesture activation. When the used recognition algorithm is a discrete gesture recognition algorithm (see 3.2), the callback shall be triggered once when the starting instruction provided by the input device is obtained; and then it is no longer triggered; and it may be triggered again after the gesture completion or the gesture cancelation. When the used recognition algorithm is a continuous gesture recognition algorithm (see 3.3), this interface shall trigger a callback once when the input device begins to provide gesture data; and then it is no longer triggered; and it may be triggered again after any gesture is successfully recognized.

Main implementation process: This interface exists in the form of a callback function.

Under the condition of discrete recognition, it may be implemented based on the hardware events provided by the operating system. Under the condition of continuous recognition, it may be implemented based on the hardware events and gesture recognition results provided by the operating system.

Input parameters: no definition.

Output parameters: no definition.

5.3.3 Gesture completion

Number: 2-3

Description: onGestureFinished(void): void

Function: This interface is used to indicate a gesture completion. When the used recognition algorithm is a discrete gesture recognition algorithm (see 3.2), after the start of the gesture, the callback is triggered when the end command provided by the input device is obtained. When the used recognition algorithm is a continuous gesture recognition algorithm (see 3.3), after the start of the gesture, a callback is triggered when any gesture is successfully recognized.

Main implementation process: This interface exists in the form of a callback function.

Under discrete recognition conditions, it may be implemented based on hardware events provided by the operating system. Under continuous recognition conditions, it may be implemented based on gesture recognition results.

Input parameters: no definition.

Output parameters: no definition.

5.3.4 Gesture cancelation

Function: The gesture interaction system shall provide at least one feedback method among visual, auditory, tactile, etc., and notify the user of the execution result after the gesture input.

The main realization process: determine the gesture recognition results; and use the hardware supported by the system to provide corresponding feedback for each recognition result.

EXAMPLE: When gesture recognition fails (not successfully recognized any gesture in the gesture set), the drive speaker emits a "deng" failure prompt tone.

Input parameters: gestureId, recognized gesture identification, see 6.7.

Output parameters: return value, etc., see 6.10.

5.4.3 System function prompt

Number: 9-3

Description: showSysGestureCommand(in gestureSet: GestureSet, in commandSet:

CommandSet): void

Function: The gesture interaction system shall provide at least one feedback method among visual, auditory, tactile, etc., and prompt the user to perform operations on the current system and corresponding gestures.

Main realization process: according to the contents of the gesture set and the application gesture instruction set, use the hardware supported by the system to prompt the operable application gesture instructions and the corresponding gestures.

EXAMPLE: Display the currently operable application gesture instructions in the form of an icon in a suitable position on the software interface, and display the gesture corresponding to each gesture instruction next to its icon.

Input parameters: gestureSet, gesture set, see 6.5; commandSet, gesture instruction set, see 6.8.

Output parameters: no definition.

5.5 System service interface

5.5.1 Create gesture set

Number: 11-1

Description: createGestureSet(out gestureSet: GestureSet): int

Function: Create an empty gesture set.

Function: Remove the existing gesture from the gesture set. After calling this interface, the removed gesture shall not be recognized.

Input parameters: gestureSet, gesture set, see 6.5; gestureId, identification of the removed gesture, see 6.7.

Output parameters: return value, etc., see 6.10.

5.5.5 Initialize gesture recognition

Number: 4-1

Description: initGestureRecognizer(in recognitionType: int): int

Function: This interface is used to initialize the gesture recognition module, specify the recognition type, and import gesture recognition module resources, etc. This interface shall be paired with the exit gesture recognition interface (see 5.5.6). The parameters of the recognition module may be customized; and the valid range of the parameters is from the start of initialize gesture recognition module to the end of the exit gesture recognition module.

Input parameters: recognitionType, identification of recognition algorithm type, see 6.2.

Output parameters: return value, etc., see 6.10.

5.5.6 Exit gesture recognition

Number: 4-2

Description: destoryGestureRecognizer(void): int

Function: This interface is used to close the currently operating gesture recognition module and release all system resources, gesture sets, and pending gesture data, etc.

This interface shall be paired with the initialize gesture recognition interface (see 5.5.5).

After exiting the gesture recognition module, if it is required to restart the gesture recognition, then call the initialize gesture recognition module and set the gesture set again.

Input parameters: no definition.

Output parameters: return value, etc., see 6.10.

5.5.7 Set recognition algorithm

Number: 4-3

Description: setRecognitionAlgorithm(in recognitionAlgorithm: RecognitionAlgorithm):

int

Output parameters: return value, etc., see 6.10.

5.5.11 Send gesture instruction decision

Number: 7-1

Description: onCommandDecisionSend(in commandId:int) :int

Function: After the interaction decision module makes the control instruction decision, it notifies the application through this interface so that the application can execute the custom instruction.

Main implementation process: This interface exists in the form of a callback function.

Whenever the interaction decision module matches the gesture recognition result output by the gesture recognition module with the application-defined instruction, the callback function is called.

Input parameters: commandId, identification of gesture instruction, see 6.9.

Output parameters: return value, etc., see 6.10.

5.5.12 Bind gesture with gesture instruction

Number: 7-2

Description: bindGesture(in gestureSet: GestureSet, in gestureId: int, in commandId:

int): int

Function: Bind the gesture in the gesture set with the application gesture instruction.

When a gesture is successfully recognized by the gesture interaction system, the application gesture instruction for binding shall be executed.

Input parameters: gestureSet, gesture set, see 6.5; gestureId, gesture identification, see 6.7; commandId, identification of gesture instruction, see 6.9.

Output parameters: return value, etc., see 6.10.

5.5.13 Application execution result

Number: 8-1

Description: onShowSysResultFeedback(in gestureId: int): int

Function: When the gesture interaction system outputs the recognition result, notify the application through this interface, so that the application can issue a custom feedback command.

Main implementation process: This interface exists in the form of a callback function.

Use interface: 11-3.

6.7 Gesture identification

Parameter name: gestureId

Value: integer, the value is mainly: -1 -- no gesture is successfully matched, 1-N -- the index of the gesture in the gesture set containing N gestures.

Description: after the recognition module recognizes the input gesture data, the ID of gesture matching in the gesture set.

Use interface: 9-2, 11-4, 7-2, 8-1.

6.8 Gesture instruction set

Parameter name: commandSet

Value: object, any an instruction set.

Description: the gesture instruction set that is customized by the application according to specific business logic, see 6.9.

Use interface: 9-3, 8-2.

6.9 Gesture instruction identification

Parameter name: commandId

Value: integer, the application can customize the value according to the specific business logic.

Description: uniquely identify an instruction in the application.

EXAMPLE: The identification of copy, paste, and the like instructions in text editing, which is used to bind gestures and instructions in the gesture interaction decision module.

Use interface: 7-1, 7-2.

6.10 Return value

Parameter name: returnValue

Value: integer, the value is 0, -1 to -5.

Description: it is used to indicate the execution result of the interface or the cause for the execution exception. For values and corresponding causes, see Table 4.

g) The gesture recognition module calls the internal interface to notify the feedback module of the gesture execution status;

h) The input device calls interface 2-3 to notify the gesture recognition module of the gesture completion;

i) The gesture recognition module calls the internal interface to notify the feedback module of the gesture completion message;

j) The feedback module calls interface 9-1 to notify the output device of system execution status;

k) The gesture recognition module calls the internal interface to recognize the command gesture;

l) The gesture set module calls the internal interface to provide a gesture template to the gesture recognition module;

m) The gesture recognition module calls the internal interface to notify the feedback module of the gesture recognition result;

n) The feedback module calls the internal interface to perform feedback information integration;

o) The gesture recognition module calls the internal interface to notify the interaction decision module of the gesture recognition result;

p) The interaction decision module calls the internal interface to match the instructions, and calls the interface 7-1 to notify the application of gesture instruction decision;

q) The application calls the internal interface to execute the application business logic;

r) The application calls interface 8-1 to notify the feedback module of the application execution result;

s) The feedback module calls interface 9-2 to notify the output device of system execution result;

t) Repeat a) to s) until the gesture interaction system exits.

7.2 Example of calling process of continuous gesture recognition interface

The general flow of continuous gesture recognition is shown in Figure 2.

It contains interfaces 2-1, 4-5, 4-6, 7-1, 8-1, 8-2, 9-1, 9-2, 9-3 and 11 internals marked with "*" Interface (not defined in this Part).

In this identification process, the interfaces for each module to execute the corresponding function calling are as follows. The sequence of each description shows the time sequence of the activities in the process:

a) First, the application calls interface 4-5 to notify the gesture recognition module to start the continuous gesture recognition;

b) The application calls interface 8-2 to notify the feedback module to execute the application function prompt;

c) The feedback module calls interface 9-3 to notify the output device to execute system function prompt;

d) The input device calls interface 2-1 to notify the gesture recognition module of data collection;

e) The system enters the cycle of continuous gesture recognition, and the gesture recognition module calls the internal interface to recognize the continuous gesture;

f) The gesture set module calls the internal interface to provide a gesture template to the gesture recognition module;

g) The gesture recognition module calls the internal interface to notify the feedback module of the gesture activation message;

h) The feedback module calls the internal interface to execute feedback information integration;

i) The gesture recognition module calls the internal interface to notify the feedback module of the gesture execution status;

j) Call the internal interface to notify the feedback module of the gesture completion message;

k) The feedback module calls interface 9-1 to notify the output device of the system execution status;

l) When the gesture recognition module calls the internal interface to notify the