Handwriting Recognition API

1. Introduction

• This section is non-normative. *

Handwriting inputs are drawings. A drawing captures the information required to recreate human’s pen-tip movements digitally.

The API proposed here aims to expose operating system capabilities to the Web. We expect handwriting recognition capabilities to vary depending on the operating system, so the API aims to achieve a flexible design that can easily integrate with operating system specific features.

We expect user agents to convert Web API data structure (defined in this spec) to the ones available on the host operating system, and connect the Web API with operating system APIs.

The API doesn’t attempt to define recognition that behaves the same on all platforms.

1.1. Definitions

• A drawing consists of multiple strokes (e.g. the above letter E consists of three strokes).

• A stroke represents one continuous pen-tip movement that happens in a time period (e.g. from one touchstart to its corresponding touchend event). The movement trajectory is represented by a series of points.

• A point is an observation of the pen-tip in space and time. It records the timestamp and position of the pen-tip on the writing surface (e.g. a touchmove event).

• A transcription is a string of Unicode characters that represents the text written in a drawing (e.g. string "WEB").

A handwriting recognizer is an interface (usually implemented by an external application or a service) that:

• Takes a drawing as input

• Output several transcriptions of the drawing

• Optionally, output segmentation information about each transcription

What constitutes a handwriting recognizer is at the discretion of the user agent.

To convert data into and from a suitable format for handwriting recognizer, the user agent should match what’s defined in this spec to equivalent concepts used in handwriting recognizer.

A handwriting recognizer may output extra information to help web applications better process the handwriting (e.g. delete a character from the handwriting).

Segmentation maps graphemes (user-perceived character) to their composing strokes and points. A grapheme can span multiple Unicode code points.

Take the handwritten text "int" for example:

• Stroke 1 and 5 makes up letter "i"

• Stroke 2 makes up letter "n"

• Stroke 3 and 4 makes up letter "t"

2. API Idioms

The task source mentioned in this specification is the handwriting recognition task source.

When an algorithm queues a Handwriting Recognition API task T, the user agent MUST queue a global task T on the handwriting recognition task source using the global object of the current realm record.

Unless specified, the realm for JavaScript objects constructed by algorithm steps is the current realm record.

3. Feature Query

Feature query interface provides allows web applications to query implementation-specific capabilities, so they can decide whether to use its feature.

const modelConstraint = { languages: ['zh-CN', 'en'] };
const modelDesc = await navigator.queryHandwritingRecognizer(modelConstraint);

// \`modelDesc\` describes the handwriting recognizer meeting the \`modelConstraint\`.
// If the constraints can't be satisfied, \`modelDesc\` will be null.
{
  textAlternatives: true,
  textSegmentation: true,
  hints: {
    alternatives: true,
    textContext: true,
    inputTypes: ['mouse', 'touch', 'stylus']
  }
}

[SecureContext]
partial interface Navigator {
  Promise<HandwritingRecognizerQueryResult?>
      queryHandwritingRecognizer(HandwritingModelConstraint constraint);
};

dictionary HandwritingModelConstraint {
  required sequence<DOMString> languages;
};

dictionary HandwritingRecognizerQueryResult {
  boolean textAlternatives;
  boolean textSegmentation;
  HandwritingHintsQueryResult hints;
};

dictionary HandwritingHintsQueryResult {
  sequence<HandwritingRecognitionType> recognitionType;
  sequence<HandwritingInputType> inputType;
  boolean textContext;
  boolean alternatives;
};

enum HandwritingRecognitionType{
  "text", "per-character"
};

enum HandwritingInputType {
  "mouse", "stylus", "touch"
};

3.1. queryHandwritingRecognizer(constraint)

This method offers web applications a way to query the underlying recognizer’s capability and decide whether they want to use the recognizer:

• If the constraint can be satisfied, resolves to a description of the handwriting recognizer,

• if the constraint can’t be satisfied, resolves to null

The same HandwritingModelConstraint can be used to invoke createHandwritingRecognizer(constraint) to create a HandwritingRecognizer that satisfies the constraint.

3.2. HandwritingModelConstraint attributes

This describes the constraint that must be satisfied by the underlying handwriting recognizer (if it will be created).

This is also used to create a handwriting recognizer in createHandwritingRecognizer(constraint).

languages

A list of [BCP47] language tags that describes the languages that the recognizer has to recognize.

If more than one language is provided, the recognizer has to recognize all of them to satisfy the constraint.

User agents should consider all possible scripts of a given language tag. For example, a handwriting recognizer that only recognizes Azerbaijani in Latin alphabet ("az-Latn") shouldn’t be used for "az" language tag, because Azerbaijani can also be written in Cyrillic alphabet ("az-Cyrl").

Consider using the most specific language tag when the distinction between scripts matters. For example, use "az-Latn" if the application only needs to work with Azerbaijani in latin scripts.

Some recognizers only work with a single language. Consider creating one recognizer for each language for better interoperability.

3.3. HandwritingRecognizerQueryResult attributes

This describes intrinsic features of a handwriting recognizer implementation.

textAlternatives

A boolean indicating whether the implementation returns multiple candidates transcriptions instead of a single one.

textSegmentation

A boolean indicating whether the implementation returns segmentation information for each transcription.

hints

A HandwritingHintsQueryResult object that describes acceptable hints in startDrawing().

3.3.1. HandwritingHintsQueryResult attributes

This describes a set of hints that can be optionally provided to startDrawing() to improve accuracy or performance.

Conventionally, the attribute names here matches the ones accepted in startDrawing() method.

recognitionType

A list of HandwritingRecognitionType enums describing the type of text that is likely to be drawn.

Hints don’t guarantee the result transcriptions meet the description.

"text"

Free form text in typical writing prose. It means the drawing represents real words. For example, a sentence in everyday speech.

"per-character"

The handwriting is made up of individual, unrelated graphemes (user-perceived characters). For example, serial numbers, license keys.

inputType

A list of HandwritingInputType enums describing how the drawing is made.

"touch"

Drawn with finger movements.

"stylus"

Drawn with a stylus.

"mouse"

Drawn with a mouse cursor.

textContext

A boolean indicating if textContext is accepted. textContext is a string that contains the text shown to user, or previously recognized text that comes before the current drawing.

alternatives

A boolean indicating if the number of alternative transcriptions can be set. This limits the maximum number of alternatives returned in getPrediction.

4. Create a handwriting recognizer

A HandwritingRecognizer manages the resources necessary for performing recognitions.

const modelConstraint = { languages: ['en'] };

try {
  const recognizer = await navigator.createHandwritingRecognizer(modelConstraint);
  // Use \`recognizer\` to perform recognitions.
} catch (err) {
  // The provided model constraint can't be satisfied.
}

[SecureContext]
partial interface Navigator {
  Promise<HandwritingRecognizer>
      createHandwritingRecognizer(HandwritingModelConstraint constraint);
};

4.1. createHandwritingRecognizer(constraint) method

• If the constraint can be satisfied, and there’s sufficient resource on the operating system to perform recognition, resolves to a HandwritingRecognizer object.

• Otherwise, rejects with an error.

The user agent might ask the user to install handwriting models and download handwriting models. Web applications shouldn’t assume this method always resolves quickly.