阅读说明：本技术 对象特性的有条件的显示 (Conditional display of object properties ) 是由 陈宇 许闻怡 于 2019-03-11 设计创作，主要内容包括：一种装置,包括用于接收指示对象和相机的相对布置的信息的部件；根据该信息确定与对象和相机的相对布置相关联的条件是否被满足；以及如果条件被满足,则使得显示对象的至少一个检测到的特性的指示。(An apparatus comprising means for receiving information indicative of a relative arrangement of an object and a camera; determining from the information whether a condition associated with the relative arrangement of the object and the camera is satisfied; and if the condition is satisfied, causing an indication of at least one detected characteristic of the display object to be displayed.)

1. An apparatus comprising means for:

receiving information indicative of a relative arrangement of the object and the camera;

determining from the information whether a condition associated with the relative arrangement of the object and the camera is satisfied; and

causing display of an indication of at least one detected characteristic of the object if the condition is satisfied.

2. The apparatus of claim 1, comprising means for: if the condition is not satisfied, causing display of image data captured by the camera without the indication.

3. The apparatus of claim 2, wherein when the condition is satisfied, an indication of at least one detected characteristic is displayed in place of at least a portion of the displayed image data.

4. The apparatus of claim 1, 2 or 3, wherein the subject comprises a human head.

5. An apparatus as claimed in any preceding claim, wherein the detected characteristic is dependent on a facial expression.

6. The apparatus of any preceding claim, comprising means for: if the condition is satisfied, causing display of an indication of a plurality of detected characteristics of the object.

7. The apparatus of claim 6, wherein the plurality of detected characteristics are associated with a plurality of different features of the face and/or body.

8. The apparatus of any preceding claim, wherein the relative arrangement comprises a relative position and/or a relative orientation.

9. The apparatus of claim 8, wherein satisfaction of the condition depends on whether at least a portion of the object is positioned outside a field of view of the camera.

10. The apparatus of claim 8 or 9, wherein satisfaction of the condition depends on whether the object is oriented facing away from the camera.

11. The device of claim 10, wherein satisfaction of the condition depends on whether an orientation of the object is at least upwardly tilted relative to an optical axis associated with the camera.

12. The apparatus of any preceding claim, wherein the information indicative of the relative arrangement is from at least one first sensor configured to detect a change in the relative arrangement.

13. The apparatus of claim 12, wherein the at least one first sensor comprises an inertial measurement unit and/or an imaging sensor.

14. The device according to any of the preceding claims, comprising means for: detecting the characteristic of the object from information indicative of the characteristic received from at least one second sensor.

15. The apparatus of claim 14, wherein the at least one second sensor comprises at least one wearable sensor.

16. An apparatus according to claim 14 or 15, wherein the at least one second sensor is configured to detect the effect of muscle movement on a measured object.

17. The device according to claim 14, 15 or 16, wherein the at least one second sensor comprises a force sensor and/or a bending sensor and/or a proximity sensor and/or a capacitance sensor and/or an inertial measurement unit and/or an electromyography sensor.

18. The apparatus of any preceding claim, wherein the displayed indication is manipulated based on a reconstruction of the object and based on at least one of the at least one detected characteristic.

19. The apparatus of claim 18, wherein the reconstruction of the object is based on a pre-captured image of the object.

20. The apparatus of claim 18 or 19, wherein the manipulation is based on at least one of the at least one detected characteristic and machine learning.

21. The apparatus of any preceding claim, comprising means for: periodically updating the displayed indication during a video communication session, wherein the indication is communicated between devices.

22. An apparatus comprising the device of any preceding claim and the camera.

23. A system comprising an apparatus according to any preceding claim and at least one of:

a camera according to any preceding claim;

at least one of the first sensors of claim 12 or 13; or

The at least one second sensor of any one of claims 14 to 17.

24. A method, comprising:

receiving information indicative of a relative arrangement of the object and the camera;

determining from the information whether a condition associated with the relative arrangement of the object and the camera is satisfied; and

causing display of an indication of at least one detected characteristic of the object if the condition is satisfied.

25. A computer program which, when run on a computer, performs the following:

causing receipt of information indicative of a relative arrangement of the object and the camera;

causing a determination from the information whether a condition associated with the relative arrangement of the object and the camera is satisfied; and

causing display of an indication of at least one detected characteristic of the object if the condition is satisfied.

Technical Field

Embodiments of the present disclosure relate to conditional display of object properties. Some embodiments relate to the conditional display of detected facial characteristics, the condition being in relation to the relative arrangement of the user's head and camera.

Background

When a user positions a camera to capture an image of a stationary or moving object, such as their own head or the head of another person, the camera mount (e.g., the user's hand) may shake, which may cause the camera to shake. Furthermore, the user or the camera support may inadvertently move such that the object is no longer positioned within the field of view of the camera or is oriented away from the camera. Some image jitter problems can be solved using anti-shake lenses or image stabilization techniques. However, this technique is only able to handle relatively minor jitter and cannot handle the problems associated with the wrong relative arrangement of the head and camera. If the relative arrangement is wrong, useful characteristics of the object, such as facial expression, cannot be easily determined from the captured image.

Disclosure of Invention

According to various, but not necessarily all, embodiments there is provided an apparatus comprising means for: receiving information indicative of a relative arrangement of the object and the camera; determining from the information whether a condition associated with the relative arrangement of the object and the camera is satisfied; and if the condition is satisfied, causing an indication of at least one detected characteristic of the display object to be displayed.

In some, but not necessarily all, embodiments, an apparatus includes means for: if the condition is not satisfied, the image data captured by the camera is caused to be displayed without indication.

In some, but not necessarily all, embodiments, if the condition is satisfied, an indication of the at least one detected characteristic is displayed in place of at least a portion of the displayed image data.

In some, but not necessarily all, embodiments, the object includes a human head.

In some, but not necessarily all, embodiments, the detected characteristics depend on facial expressions.

In some, but not necessarily all, embodiments, an apparatus includes means for: if the condition is satisfied, an indication of a plurality of detected characteristics of the display object is caused.

In some, but not necessarily all, embodiments, the plurality of detected characteristics are associated with a plurality of different features of the face and/or body.

In some, but not necessarily all, embodiments, the relative arrangement includes relative position and/or relative orientation.

In some, but not necessarily all, embodiments, satisfaction of the condition depends on whether at least a portion of the object is positioned outside of the field of view of the camera.

In some, but not necessarily all, embodiments, the satisfaction of the condition depends on whether the object is oriented away from the camera.

In some, but not necessarily all, embodiments, satisfaction of a condition depends on whether the direction of the object is at least upwardly tilted relative to an optical axis associated with the camera.

In some, but not necessarily all, embodiments, the information indicative of relative arrangement is from at least one first sensor configured to detect a change in relative arrangement.

In some, but not necessarily all, embodiments, the at least one first sensor comprises an inertial measurement unit and/or an imaging sensor.

In some, but not necessarily all, embodiments, an apparatus includes means for: a characteristic of the object is detected from information indicative of the characteristic received from the at least one second sensor.

In some, but not necessarily all, embodiments, the at least one second sensor comprises at least one wearable sensor.

In some, but not necessarily all, embodiments, the at least one second sensor is configured to detect an effect of muscle movement on the measured object.

In some, but not necessarily all, embodiments, the at least one second sensor comprises a force sensor and/or a bending sensor and/or a proximity sensor and/or a capacitive sensor and/or an inertial measurement unit and/or an electromyography sensor.

In some, but not necessarily all, embodiments, the displayed indication is manipulated based on the reconstruction of the object and based on at least one of the at least one detected characteristic.

In some, but not necessarily all, embodiments, the reconstruction of the object is based on a pre-captured image of the object.

In some, but not necessarily all, embodiments, the manipulation is based on at least one of the at least one detected characteristic and machine learning.

In some, but not necessarily all, embodiments, an apparatus includes means for: the displayed indication is periodically updated during the video communication session, wherein the indication is communicated between the devices.

According to various, but not necessarily all, embodiments there is provided an apparatus comprising a device and a camera.

According to various, but not necessarily all, embodiments there is provided a system comprising an apparatus and at least one of: a camera; at least one first sensor; or at least one second sensor.

According to various, but not necessarily all, embodiments there is provided a method comprising:

receiving information indicative of a relative arrangement of the object and the camera;

determining from the information whether a condition associated with the relative arrangement of the object and the camera is satisfied; and

if the condition is satisfied, causing an indication of at least one detected characteristic of the display object to be displayed.

According to various, but not necessarily all, embodiments there is provided a computer program that, when run on a computer, performs:

causing receipt of information indicative of a relative arrangement of the object and the camera;

causing a determination, from the information, whether a condition associated with the relative arrangement of the object and the camera is satisfied; and

if the condition is satisfied, causing an indication of at least one detected characteristic of the display object to be displayed.

According to various, but not necessarily all, embodiments, there are provided examples as claimed in the appended claims.

Drawings

Some example embodiments will now be described with reference to the accompanying drawings, in which:

FIG. 1 illustrates an example of a method;

FIG. 2 illustrates an example of a system and device;

FIG. 3 illustrates an example of an object in a first relative position and a first relative orientation, and FIG. 3B illustrates an example of an object in a second relative orientation; and figure 3C illustrates an example of an object in a second relative position;

fig. 4A illustrates an example of a human head in a first relative orientation, fig. 4B illustrates an example of a human head in a second relative orientation, fig. 4C illustrates an example of camera image data of a human head in a first relative orientation, and fig. 4D illustrates an example of an indication of at least one detected characteristic of a human head;

FIG. 5 illustrates an example of a method;

FIG. 6 illustrates facial expressions of a human head;

FIG. 7 illustrates an example headset including a second sensor;

FIG. 8 illustrates an example of eyewear including a second sensor;

FIG. 9A illustrates an example of a head showing the motion dependence of feature points, and FIG. 9B illustrates an example of a geometric model describing the motion dependence of feature points;

FIG. 10A illustrates an example of a head reconstruction showing feature points highlighted for manipulation, and FIG. 10B illustrates an example of a head reconstruction after manipulation; and

fig. 11A illustrates an example of an apparatus, and fig. 11B illustrates an example of a computer-readable storage medium.

Detailed Description

Referring to fig. 1-4D, fig. 1 illustrates a method 100 comprising: receiving information (110) indicative of a relative arrangement of the object and a camera 206 (imaging sensor); determining from the information whether a condition (120) associated with the relative arrangement of the object and the camera 206 is satisfied; if the condition is satisfied, an indication of at least one detected feature of the display object is caused 406 (130). Alternatively, if the condition is not met, image data 402 captured by the camera 206 is displayed without the indication 406 (block 140).

In the examples disclosed below, but not necessarily in all examples, the object is a human head of a user of the camera 206. In other examples, the object may be the head of a human other than the user of the camera 206, or an animal head.

In the examples disclosed below, but not necessarily in all examples, camera 206 is configured as a video camera. When the method 100 is performed, the camera 206 captures moving images. Additionally or alternatively, the camera 206 may be configured to capture still images. The camera 206 may be a visible light camera or may be configured to image other wavelengths in the electromagnetic spectrum.

Example implementations of the method 100 are described below with particular reference to fig. 1-4D.

Block 110 is described in more detail. Block 110 includes receiving information indicating a relative arrangement of the header 216 and the camera 206.

When the camera 206 is set to capture an image of the user's head 216, the relative arrangement of the camera 206 and the head 216 may temporarily change. At times, their head 216 may be positioned outside the field of view 222 of the camera (less than 360 degrees). Sometimes, their head 216 may be too close or too far from the camera 206. Sometimes, their head 216 may be imaged at a sub-optimal angle such that the user's face may be unclear. Sometimes, a user may become tired from lifting the camera 206 (e.g., a handheld camera), and their arms may droop such that the user's head 216 is not parallel to the optical axis 224 of the camera 206 and is imaged at a suboptimal angle. In a further example, the user may wish to multitask in a manner that requires them to exit the field of view 222 of the camera 206.

The above-described difficulty in maintaining the desired relative arrangement may prevent camera 206 from capturing subtle emotions, such as those conveyed by facial expressions. Emotions can deliver significantly more content than speech, which represents a potential advantage of camera communication over text-based communication.

The method 100 of fig. 1 advantageously provides an improved image of the user and may more reliably provide background information regarding characteristics of the user's head, such as facial expressions.

Accordingly, information indicative of the relative arrangement of the head 216 and the camera 206 is received at block 110 to monitor the relative arrangement. The monitoring may be performed automatically.

In some, but not necessarily all, examples, the information indicative of relative arrangement is from at least one first sensor configured to detect a change in relative arrangement.

The at least one first sensor is a sensor selected from a first group of one or more sensors. The first set of sensors may include an inertial measurement unit. The inertial measurement unit may include an accelerometer and/or a gyroscope. One inertial measurement unit corresponds to one sensing mode and one measurement axis. The first set of sensors may include a plurality of inertial measurement units defining a plurality of sensing modes and/or a plurality of sensing axes. The inertial measurement unit may include a three-axis accelerometer and/or a three-axis gyroscope. The first set of sensors may include an imaging sensor such as the camera 206 described above (or another camera). The imaging sensor may be a 3D imaging sensor, such as a stereo camera or plenoptic camera, or may be a 2D imaging sensor.

The first set of sensors may include sensors 212 on the same device as the camera 206, see, e.g., fig. 2, which shows the device 202 including the camera 206 and the first sensors 212. Additionally or alternatively, the first set of sensors may include sensors on the head 216 and/or body of the user positioned to enable monitoring of the relative arrangement.

The received information may be processed to determine relative placement, defined as relative position and/or orientation. In some, but not necessarily all examples, the process may perform dead reckoning (dead reckoning). If the received information includes image data, a head gesture recognition algorithm may be applied.

To be able to accurately determine the orientation, the orientation of the first sensor relative to the host camera device and/or the user may be constant, e.g. such that the inertial measurement signals are consistent. If the received information includes image data, eye gaze tracking and/or head pose estimation may be used to detect the direction of the user's head 216. Head pose estimation is a more accurate indicator of sub-optimal head direction compared to eye gaze tracking.

Once the information is received at block 110, the method 100 proceeds to block 120, described below.

Block 120 includes determining from the information whether a condition associated with the relative arrangement of the head 216 and the camera 206 is satisfied. This condition may be related to acceptable deviation from the relative arrangement. In some, but not necessarily all examples, determining whether the relative arrangement is acceptable for deviation from the acceptable relative arrangement may depend on whether the deviation from the reference relative arrangement exceeds a threshold. The reference relative arrangement may include reference to an acceptable relative positioning, such as the head being centered in the field of view 222 of the camera. Additionally or alternatively, referencing a relative arrangement may include referencing an acceptable relative orientation, such as the head direction being parallel to the optical axis 224. The threshold may be configured to allow some deviation of the relative arrangement from the reference relative arrangement if the condition is not met. In other examples, the determination of acceptability may depend on whether certain facial features are identified by image analysis of the image data from camera 206. Examples of threshold-based and feature-tracking approaches are described below.

Fig. 3A to 3C represent three different relative arrangements, two of which satisfy the condition and one of which does not. Fig. 4A and 4B represent two different relative arrangements, one of which satisfies the condition and one of which does not satisfy the condition.

In fig. 3A and 4A, the relative arrangement of the head 216 and the camera 206 results in a first relative position P1 and a first relative orientation O1. The first relative position is within the field of view 222 of the camera 206. The first relative orientation is substantially parallel to the optical axis 224 of the camera 206, and the head 216 is looking toward the camera 206 (i.e., a front orientation).

Satisfaction of the condition may depend on whether at least a portion of the head 216 is determined to be positioned outside the field of view 222. The first relative position in fig. 3A does not satisfy the condition because the head 216 is within the field of view 222 and is facing frontally toward the camera 206 to provide a clear view of the facial features.

In order for the relative positioning to satisfy the condition, the relative positioning may need to be changed such that at least a portion of the head 216 is positioned outside the field of view 222 as a result of moving the camera 206 and/or the head 216. For example, in fig. 3C, the head 216 is in a second position P2 relative to the camera 206 such that at least a portion of the head 216 is out of the field of view 222 such that the condition is satisfied.

In some examples, satisfaction of the condition may require that the entire head 216 exit the field of view 222. In other examples, satisfaction of the condition may require that at least a portion of the head 216 exit the field of view 222, such as more than 50% of the head 216 exit the field of view 222. In some examples, the head 216 is moved closer to the camera 206 such that the head 216 is cropped at the edge of the field of view 222 without affecting whether the condition is satisfied. The satisfaction of the condition may be determined based on the centering of the head 216 in the field of view 222. For example, satisfaction of the condition may depend on whether a tracking reference location (e.g., center) of the head 216 exits the field of view 222 or enters within a threshold distance of an edge of the field of view 222, or whether an identified facial feature of the head 216 that is capable of expressing emotion (e.g., mouth, eyebrows) exits the field of view 222. If the head 216 moves off-center, the likelihood of satisfying the condition may increase.

In fig. 3B and 4B, the head 216 and the camera 206 are in a second relative orientation O2. The second relative orientation in fig. 3B satisfies this condition because the head 216 points away from the camera 206 (non-frontally oriented) while the head 216 is still within the field of view 222 of the camera 206. Non-positive relative orientations hide emotional context.

In some examples, satisfaction of the condition depends on whether the relative orientation exceeds a threshold. The threshold may be selected from a range of greater than 0 degrees from the optical axis 224 to about 90 degrees from the optical axis 224. In some examples, the range may be from about 20 degrees to about 60 degrees. This will reduce false-positive satisfaction of the condition if the user only glances around, and ensure that the condition is satisfied when the facial features are no longer sharp.

In some examples, the threshold may depend on which axis the change in relative orientation occurs. For example, if the user looks up such that the orientation is "from below," the threshold may be lower than for a viewing angle that is "from the side. This is because emotional context may be more difficult to infer in a "from below" perspective than in a sideways perspective. Furthermore, the "from below" perspective is considered to be an unsightly perspective. In some examples, satisfaction of the condition depends on whether the direction of the head 216 is at least tilted (oriented) upward relative to the optical axis 224. If the head 216 is not tilted up, e.g., down, the condition may not be satisfied.

The fulfilment of the condition may be determined according to the relative arrangement at the instant and optionally according to the relative arrangement in the past. For example, if the head 216 is not in the field of view 222 from the beginning, the condition may not be satisfied.

To reduce false positives caused by small movements such as camera shake, satisfaction of the conditions may require that unacceptable relative placement occur for a duration and/or at a frequency above the threshold.

In the above example, the condition may be satisfied by a change in relative position alone, and may be satisfied based on a change in relative orientation alone. In other examples, satisfaction of the condition depends on one of the relative position or relative orientation rather than the other. In a further example, the condition may only be satisfied by a combination of relative arrangement and relative orientation, rather than by relative arrangement or relative orientation alone.

Satisfaction of the condition is necessary, and optionally sufficient, to proceed to block 130 as described below.

Block 130 of fig. 1 includes causing an indication 406 of at least one detected characteristic of the display header 216 to be displayed if the condition is satisfied. The detected characteristics may depend on the facial expression. Facial expressions provide emotional context. Facial expressions are personal to the user and therefore represent their respective characteristics. Thus, despite the suboptimal relative arrangement of the head 216 and the camera 206, emotion context can be advantageously conveyed. Displaying detected characteristics indicative of the user's actual expression conveys the emotional context more clearly than, for example, displaying undetected characteristics indicative of a general smile (e.g., an avatar or emoticon). A detailed discussion of how the indication 406 is determined and how the characteristic is detected is provided later.

An indication of the current characteristic detected is shown in fig. 4D. An indication 406 of a smile and raised eyebrows has been detected and is indicated by the smile and raised eyebrows in the representation 404.

The display displaying the indication 406 of fig. 4D may be local to the camera 206, such as the display 208 of the device 202 of fig. 2. Additionally or alternatively, the display on which the indication 406 is displayed may be a display remote from the camera 206, such as a display of a remote device for receiving image data as part of a video communication session. This helps remote third party users to infer the user's emotions when they cannot see the user.

In some, but not necessarily all examples, the representation 404 of fig. 4D is a rendered reconstruction of the entire head 216 or at least a portion of the head 216, where at least a portion of the reconstruction is configured to indicate at least one of the at least one detected feature of the head 216. The term "reconstruction" will be understood to represent a model generated at least in part by a computer that is automatically constructed based on information indicative of the actual head 216 of a particular user and configured to be rendered on the display 208. The reconstruction may be based on a 2D or 3D computer model of the actual head 216 of the particular user. The effect of using the reconstruction is that the user's facial expressions and emotions are conveyed in a more accurate and familiar manner than if the detected features were indicated in a simpler manner.

In a simpler example where no reconstruction is required, the indication 406 may instead include a content item such as text, a symbol, or a pre-captured camera image (e.g., a pre-captured photograph or avatar associated with the detected characteristic). The content items may be selected using an emotion classification algorithm for associating at least one detected characteristic with a particular content item of the plurality of selectable content items associated with a different detected characteristic.

FIG. 4D illustrates a representation 404 of the user's head 216 that may be displayed as a result of block 130. In this example, the user's head 216 has a relative orientation as shown in fig. 4B and 3B, which results in the satisfaction of the condition. Fig. 4D may look the same or similar if the relative positioning is the reason for the condition being met, rather than the relative orientation.

The representation 404 may include features of the head 216 that are not indicative of the detected current facial expression characteristics, such as hair and skin texture. Those features may be displayed from a pre-captured image of the user's head 216, or may be displayed from an avatar or placeholder image.

If the conditions of block 120 are not met, block 140 may be performed instead of block 130.

Block 140 includes causing output of different data without causing display of the indication 406 if the condition is not satisfied. In an example, the output is a display output and the different data includes image data 402 captured by the camera 206. The head 216 is likely to be clearly visible in the image data 402 because the head 216 is sufficiently inside the field of view 224 of the camera 206 and facing the camera 206, i.e., the condition is not satisfied. Fig. 4C shows an example of image data 402 captured by camera 206 for block 140. Fig. 4C shows the user's head 216 of fig. 3A captured by the camera 206 in real-time. The current facial expression of the user is therefore represented in fig. 4C.

As an alternative to the above-described implementation of block 140, the different outputs may include audio or tactile feedback, for example to prompt the user to move their head 216 and/or camera 206 to a different relative arrangement such that the condition is no longer satisfied. Alternatively, if the different output is a display output, the different output may include a notification or any other alert. Alternatively, block 140 may be omitted entirely and the non-satisfaction of the condition may cause the method 100 to terminate or loop to block 110.

The background of fig. 4D behind the user's head may be a static or moving image and may even be captured by the camera 206 in real-time, depending on the implementation. For example, when the condition is satisfied, the indication (block 130, fig. 4D) may replace part or all(s) of the displayed image data (block 140, fig. 4C). In one example, the replacing may include forming a composite image where the indication (and/or representation 404) is a foreground object and camera image data in the background around the foreground object remains visible (partial replacement). This may be useful if the user is talking about or reacting to what is visible in the image data from the camera 206. In another example, the display may automatically switch between displaying the indication without image data and displaying the image data without the indication, regardless of when the satisfaction state of the condition changes (global replacement). The context may be predetermined or configurable automatically and/or manually.

When the representation 404 is displayed, it may be centered as shown in FIG. 4D, or may be offset, for example, to a corner of the image. The scale of the indication may be zoomed in to better convey emotion, or zoomed out to be unobtrusive, or varied according to the detected distance of the user from the camera 206. The indication may be aligned with the user's head 216 if the user's head 216 is still in the field of view 224. The size and/or alignment of the indication may be predetermined or may be automatically and/or manually configurable.

The method 100 may be repeated periodically. The periodic repetition of the method 100 may result in a real-time feed of the displayed information. The method may be repeated to automatically update the real-time feed of the displayed indication 406 and/or to automatically update the real-time feed of the image data 402. The average time interval between each repetition of the method 100 may be less than one second.

The difference between the displayed indication 406 of the characteristic (block 130, fig. 4D) and the image data 402 of block 140 (fig. 4C) is that the image data 402 indicates content currently sensed by the camera 206, while the indication 406 does not indicate content currently sensed by the camera 206, as the characteristic is not clearly visible in the image data from the camera 206. This is due to the facial features being tilted with respect to the camera 206. The indication 406 of fig. 4D may instead indicate what is currently sensed by other sensors, as will be described below.

FIG. 5 illustrates an example of a method 500 of how a characteristic is detected and a representation 404 suitable for display is created that indicates the characteristic. The method 500 of fig. 5 may be performed only if block 120 is satisfied, or may be "always on" whether or not the condition is satisfied, such that the user may select to display the representation 404 instead of the image data 402.

At block 510, the method 500 includes receiving information indicative of head characteristics from the at least one second sensor 218. The use of sensors means that it is not necessary for the user to manually input information indicative of their emotional state. The method 500 of fig. 5 may be performed automatically and repeated periodically in accordance with the method 100 of fig. 1.

The characteristic may be detected, for example, by receiving information from the at least one second sensor 218 each time, which information ensures that the indication of the displayed characteristic indicates the current characteristic of the user. For example, a property may be a current property if the displayed indication shows the property not exceeding one minute after the user property when displayed. In some examples, the delay may not exceed one second.

At least one second sensor 218 will now be defined. The at least one second sensor 218 is a sensor selected from a second set of one or more sensors. The second set of sensors includes at least one sensor that is different from the first set of sensors and is not the camera 206. When the face is out of the lens, the camera 206 will be restricted from use.

At least some of the second sensors 218 may be configured to detect the effect of muscle movement on a measured object (measurand). In particular, at least some of the second sensors 218 may be configured to detect the effects of facial muscle movements. Facial muscle movements may include muscle tone. At least some of the second sensors 218 may be positioned in contact with or proximate to the user's head 216. The second sensor 218 may include one or more wearable sensors. The second sensor 218 may be worn when the methods of fig. 1 and 5 are performed.

The particular positioning of at least some of the second sensors 218 relative to particular facial muscles enables the detection of the associated effects of particular emotions on the motion of particular facial muscle groups. Fig. 6 shows the major facial muscles of a human head 216. Table 1 below indicates which muscles are involved in which emotions:

table 1: muscles involved in seven major human emotions

The second set of sensors may comprise force sensors and/or flexion sensors 708 and/or proximity sensors 808 and/or capacitive sensors 706 and/or inertial measurement unit 704 and/or electromyography sensors. The inertial measurement unit 704 may also be used as one of the first sensors.

The second sensor 218 may be made wearable by attaching or embedding the second sensor 218 in a wearable accessory. Accessory as described herein means a wearable device that provides at least aesthetic and/or non-medical functions. Examples of wearable accessories include headphones (or audible devices) 700 (fig. 7), eyeglasses 800 (fig. 8), and/or any other type of wearable accessory (e.g., clothing, jewelry, and hair accessories). An earphone is a wearable accessory that can be worn in or on the ear. An audible device is defined herein as a headset with an audio speaker.

The wearable accessory ensures that the second sensor 218 is worn in the desired position on the user's head. For the purposes of this disclosure, a desired position is any position on the human head that moves according to the contraction and/or relaxation of facial muscles in a manner that is detectable by the second sensor 218. Such locations include locations on the head and may also include locations in the upper region of the neck that are anatomically classified as part of the neck.

In some, but not necessarily all, examples, more than one second sensor 218 is worn on one or more wearable accessories. Wearing a plurality of second sensors 218 may include wearing second sensors 218 that provide different sensing modes.

Wearing the plurality of second sensors 218 may include wearing the second sensors 218 at different locations of the user's head for the same or different modes. In some examples, the desired position may be located on the left and right sides of the head. These locations may be on opposite sides of symmetry of the head. This provides a better distinction between symmetric and asymmetric facial expressions (e.g., smile versus half-smile). In other examples, the distribution of locations may target different facial muscles and may or may not involve symmetric positioning.

The wearable accessory including the second sensor 218 may be configured to be worn in a reusable manner. The reusable means that the wearable accessory can be removed and later re-worn without irreparable damage to the wearable accessory upon removal. The wearable accessory is wearable outside the user's body, thus not requiring implantation.

The wearable accessory including the second sensor 218 may be configured to not be disposable. For example, the wearable accessory may be configured to be friction and/or biased mounted. This eliminates the need for disposable adhesives and the like. However, in alternative implementations, the wearable accessory is configured for disposable operation, e.g., the wearable accessory may be part of an adhesive patch.

The wearable accessory may include circuitry, such as a power source and circuitry, for causing the second sensor 218 to function. The methods described herein may be performed by circuitry of a wearable accessory, or may be performed by an external device. The wearable accessory may include an interface, which may include a wire or antenna, for communicating information to an external device.

The wearable accessory may provide one or more wearable accessory functions. Examples of other functions of the wearable accessory include, but are not limited to, providing a human machine interface (input and/or output), noise cancellation, positioning additional sensors for other uses, and the like. Some wearable accessories may even include additional medical/non-accessory functions (e.g., corrective/colored eyeglass lenses, positioning health monitoring sensors). The headset 700 of fig. 7 may include an audio speaker. The eyewear 800 of fig. 8 may include colored lenses and/or corrective lenses.

The headset 700 of fig. 7 will be defined in more detail. The advantage of the headset 201 is convenience compared to e.g. wearing special clothing or unnecessary glasses. Another advantage is that the headset 201 is positioned close to several facial muscles strongly associated with common facial expressions.

Fig. 7 shows two earphones 700 defining a wearable accessory, for the left and right ear respectively. In other examples, only one earphone 700 is provided for use with only one ear.

The earphone 700 of fig. 7 is an in-ear earphone 700 for embedding in the pinna. The in-ear headphone 700 may be configured for insertion into the adjacent ear canal. The in-ear headphone 700 may be configured to be embedded in the outer ear or concha. One advantage is that there is a relatively strong correlation between the movement of facial muscles that form common facial expressions and the deformation or movement of the portion of the ear in contact with the headset 700. By positioning the second sensor 218 within the earpiece 700, this relative motion can be exploited so that the output of the second sensor depends on the motion or deformation of the ear. Thus, the headset 201, and in particular the in-ear headset 201, significantly reduces the amount of data processing required to isolate meaningful signals from signal noise as compared to other wearable accessories. Other wearable accessories may work when positioned at the various head positions specified herein and form part of the present disclosure.

The headset 700 of fig. 7 may include an inertial measurement unit (not shown) and/or an electromyography sensor (not shown). The inertial measurement unit may be configured to detect the effects of head 216 movements, such as nodding and/or shaking head 216, and/or facial muscle movements on the ears. The information from the inertial measurement unit may indicate the intensity and/or frequency of the nodding or shaking head of the head 216. The inertial measurement unit may optionally also belong to the first group of sensors, since it also enables a determination of the relative arrangement.

The headset 700 of fig. 7 may include a force sensor 702. The minimum sensitivity of the force sensor 702 may be 0.05N. The force sensor 702 may detect force from pressure on the force sensor 702 through deformation of the ear. The deformation of the ear may be due to a tension of the supraauricular and zygomatic muscles, involving fear, anger and surprise emotions.

The headset 700 of fig. 7 may include a bend sensor 708. The bend sensor 708 may be configured to detect bending of the wire 710 of the headset 700 if the headset is wired (e.g., a headset wire hanging from an ear). When the masseter, zygomatic, and buccinator muscles are under tension (happy), the face bulges, which pushes the wire 710 and causes some bending. An example of a compact bend sensor for detecting small bends in a wire is a nanosensor, which includes a torsional opto-mechanical resonator and a waveguide such as an optical cable for detecting torsion (bending).

The headset 700 of fig. 7 may include a capacitive sensor 706. Capacitive sensor 706 may be disposed in a conductor 710. As head 216 moves or the expression changes, the face may contact wire 710, causing a change in the capacitance of the wire at a location along wire 710. The capacitive sensor 706 may be used to detect happiness (smiling).

The headset 700 may be configured to maintain a predetermined orientation of the second sensor 218 relative to the user to ensure that valid data is obtained. In the example of fig. 7, the headset 700 of fig. 7 includes elements configured for engagement with the intertragic notch of a user's ear. The element may include a sleeve for the wire 710 configured to increase the effective stiffness of the wire 710 and reduce bending fatigue. If the headset 700 is wireless, this element may include an internal antenna for wireless communication or be used for other purposes than engaging the inter-tragus notch to position the headset 700 in a predetermined orientation.

One or more of the above-described second sensors 218 of the headset 700 may additionally or alternatively be part of another wearable device, such as the eyewear 800 of fig. 8.

The eyewear 800 of fig. 8 will be defined in more detail.

The eyewear 800 of fig. 8 may include one or more proximity sensors 808 and/or electromyography sensors (not shown). The proximity sensor 808 may be configured to detect a distance between the glasses and a corresponding local location on the face. When muscles (e.g., orbicularis oculi, frontalis, levator labei, nasal muscles) around the eyes and nose are tense (e.g., slight, aversion, sadness), the face around the eyes and nose may bulge and thereby change the distance between the local location on the face and the corresponding proximity sensor 808. A particular type of emotion is related to the degree of stress and thus to the amount of change in distance.

Four proximity sensors 808 are shown in fig. 8 and the proximity sensors 808 are disposed on eye wires 802, 804: two on the left eye wires 804 and two on the right eye wires 802. The proximity sensors 808 include an upper pair of proximity sensors 808 and a lower pair of proximity sensors 808. The lower pair of proximity sensors 808 is spaced apart more than the upper pair of proximity sensors 808. The lower pair may be located in the distal lower corner of the respective eye-leads 802, 804, and the upper pair may be located around the proximal upper corner of the respective eye-leads 802, 804, near the bridge 806. However, a different number of proximity sensors 808 may be provided in other examples and in different layouts.

The second set of sensors may optionally further include motion sensors (e.g., inertial measurement units) attached to the body to detect body gestures that accompany changes in facial expression, thereby improving the accuracy of emotion detection.

Once the information is received from the second sensor, the method 500 continues to detect a characteristic of the head 216 and determine the indication 406 to be displayed. Information from the plurality of second sensors 218 may be synthesized first to improve accuracy. Blocks 520 through 550 illustrate the use of techniques that result in a realistic rendering of the user's head 216 and facial expressions. However, in other examples, the indication 406 to be displayed may be simpler and not require that the actual user be represented in more detail than re-creating their detected facial expression characteristics.

In block 520, the method 500 includes determining a desired movement of at least one first feature point or a plurality of first feature points associated with the information. The desired movement of the first feature point is strongly correlated with and therefore measurable from the output of the corresponding second sensor 218, such that the indication "detected feature" as described herein may correspond to the desired movement of the first feature point.

Fig. 9A shows a header 216 including a plurality of predetermined feature points, some of which are first feature points and some of which are additional feature points. Each feature point relates to a different feature of the face and corresponds to a different face position. The spatial density of the feature points around the spatial domain corresponding to the face may be equal or variable to increase the spatial density of the feature points of the facial expression part. The 27 characteristic points are shown in FIG. 9A and are referred to herein as d1-d 27. Feature points d1-d10 are first feature points and d11-d27 are additional feature points. In other examples a different number of first feature points may be provided.

The 27 illustrated feature points move in a correlated manner according to facial muscle movements. For example, movement of the orbicularis oculi muscle is associated with movement of d1, d2, d3, d4, d12, d13, d15, d16, d17, and d 19. The movement of the orbicularis oris muscle is associated with the movement of d21, d22, d23 and d 24. The movement of the frontal muscle is associated with the movement of d11 and d 14. The movement of the cheekbones is associated with the movement of d5, d6, d18 and d 20. The movement of the lowering angle muscles is associated with the movement of d25, d26 and d 27.

Block 520 determines a desired amount of movement of the first feature point that is proportional to the value of the sensed information. The scale may be predetermined as a result of experimentation and/or may be refined using machine learning (described later). There is at least the following highly correlated association between the first characteristic points d1-d10 and the second sensor 218:

the proximity sensor 808 on the glasses 800 is associated with feature points d1, d2, d3 and d 4;

the force sensors 702 on the headset 700 are associated with feature points d5 and d 6;

the bending sensor 708 on the headset 700 is associated with feature points d7 and d 8; and

the capacitive sensor 706 of the headset 700 is associated with feature points d9 and d 10.

The method 500 then proceeds to optional block 530. Block 530 includes determining a desired amount of movement for additional feature points (e.g., d11-d27) based on the feature points (e.g., d1-d10) determined in block 520. In this example, the computer model includes an additional 17 feature points d11-d27, the additional 17 feature points d11-d27 not being directly associated with the output of the second sensor 218, but rather with the movement of the first feature points d1-d 10. The association may be determined via estimation or via machine learning as described below. For example, smiling alters the cheek and masseter muscles, which means that d21, d22, d24, and d25 may also change when d9 changes.

A mathematical model may be used to approximate the movement relationship between the feature points determined in block 520 and the additional feature points.

In block 520 and/or block 530, a predetermined 3D triangle model may be used to determine the desired movement of the additional feature points. An example of a 3D triangle model is illustrated in fig. 9B. A plurality of 3D triangular models may be provided for representing all 27 feature points.

For a particular face, the feature points d1-d27 may be divided into two groups:

group 1 (G1): d1, d2, d11-d16, d3, d4, d17 and d 19; and

group 2 (G2): d5-d10 and d 18-27.

G1 includes the first feature point from block 520 (e.g., d1, d2, d3, and d4), and some additional feature points (d11-d16, d17, and d19) associated with the first feature point are determined by machine learning. The mathematical model associates each first feature point with one or more additional feature points.

An example model for G1 is described below, associating d1 with d 11. If the observed change in sensor distance is d, the original distance of d1 from d11 is a, and the determined muscle stretch rate is w, then the observed change in distance of d1 from d11 for a given sensor, a', is:

additional feature points d12 and d13 may be associated with d1, respectively. Additional feature points d14-d16 may be associated with d2, respectively.

For G2, a triangle model may be used to describe the relationship between feature points. Fig. 9B illustrates an example triangular model representing a temporal muscle. When the temporalis muscle is tensed, the skin will move and will drive vertex V1 (the feature point) to the right in fig. 9B. This action will increase the pressure on the earpiece force sensor 702 when the edge E2 moves to the right. Thus, there is a relationship between pressure and vertex. For fig. 9B, the mathematical relationship can be derived assuming that the measured pressure is f, the muscle stretch is w, the relationship between the detected f and the facial changes follows the function g, and the measured original distance between the feature point V1 and the point of influence (additional feature point) E2 is d:

f～g(wd)

the method 500 may include means for training a machine learning algorithm, wherein the machine learning algorithm is configured to control which feature points are manipulated according to information from the second sensor and/or the extent to which they are manipulated. For example, for a given user (in-use training) or for any user (off-line training), machine learning can be used to obtain an accurate relationship between the measured data of muscle stretch rates w and d and f.

Training for a given user may include causing the display 208 or other output device of the user device to show or describe the desired facial expression that the user is attempting to and matching. The user may shoot themselves on the camera 206 to provide a training data set of measurements of the movement of their feature points for a given facial expression. In an example, image analysis may be used to determine the precise locations of the feature points. The training data set may train a neural network, such as a convolutional neural network, or any other suitable machine learning algorithm. The predicted relationship between feature points may then be refined by observation, for example to minimize a loss function.

An example loss function is defined below. For each learning period T, the inputs will be the measured point information fi ∈ F and the measured 27-point collaboration relationship<X_T，Y_T>Where F is the set of first feature points (e.g., d1-d10) of block 520. Outputting new collaboration relationships that will be points<X′_T+1，Y′_T+1>. The loss function used in the learning process may include predicted<X′_T+1，Y′_T+1>And reality<X_T+1，Y_T+1>Distance function of comparison, e.g. in mean square errorDifference or other suitable format:

machine learning may be used to improve not only the model of the G2 parameters of block 530, but also the G1 parametric relationships, and to improve the determination of the required movement of the feature points of block 520. More generally, machine learning may be used as long as the indication 406 displayed at block 130 is intended to indicate more than just the user characteristic detected directly by the second sensor.

Once the desired movement of all detected characteristics (feature points) is determined for a given sensor information, the method 500 proceeds to block 540.

At block 540, the method 500 includes manipulating feature points of the reconstruction of the face (which may be the entire head 216 or the reconstruction of only the face) based on the detected characteristics. The reconstruction may be a 3D reconstruction or alternatively a 2D reconstruction. The reconstruction may be based on a pre-captured image of the head 216, such as an image captured prior to a video communication session that may occur with the method of the present disclosure. Thus, the reconstruction may not indicate the current characteristics of the detected head prior to manipulating the reconstruction as described below. The pre-captured image may be a face with neutral, non-emotional expressions to provide the underlying data. The reconstruction may include a mesh, such as a triangular mesh, or may include voxels (voxels) or any other suitable data structure. The mesh may represent a tangential space (surface) of the reconstructed face.

Fig. 10A illustrates an example of reconstruction 10 showing one feature point to be manipulated. The feature points are associated with particular vertices 11 of the reconstruction 10. The movement of the feature point corresponds to the movement of the vertex 11. Fig. 10B illustrates an example of the reconstruction 10 of fig. 10A, but after the feature points have been manipulated. The cheek apex 11 is visibly convex relative to the same apex 11 in fig. 10A, thus indicating at least one detected feature as described herein.

In some examples, the vertex 11 is not changed if the required amount of movement of the feature point/vertex 11 is below the threshold β. The value of the threshold β may depend on the configuration resolution of the grid and/or the relief map (if provided).

For nodding and shaking movements, the manipulation may include translation or rotation of the entire reconstruction head. For changes in facial expression involving particular facial muscles, the manipulation may include distorting the tangent space via the feature points to indicate changes in facial expression of the reconstructed head.

At block 550, the method 500 includes adding further details of the human head, such as texture (e.g., skin texture, hair), color, and relief (wrinkles). This conveys the emotional state better than the use of a generic avatar. However, in some examples, the user may choose to customize their reconstruction 10 and/or its texture/relief/color.

Texture, color, and relief can be obtained from a pre-captured image of the head 216. A mathematical operation of known type is performed to map the texture/relief map onto the manipulated reconstruction 10 of the face. Known lighting effects may be applied to better illustrate details of the user's emotions, and optionally the lighting effects may be configured to recreate the detected lighting parameters in the user's actual location detected by the camera 206.

Once block 550 is complete, if the condition is satisfied, the method may proceed to block 130. In this example, block 130 may include causing display of representation 404 in the form of a manipulated reconstruction of a face having any texture and effect throughout.

All of the methods and features described above may be performed by an apparatus 204, such as the apparatus shown in FIG. 11A. The apparatus 204 may be provided in the common device 202 with the camera 206, as shown in fig. 2, or may be provided in a device separate from the device 202 that includes the camera 206.

Thus, in one example, a device 202 is provided that includes an apparatus 204 and a camera 206, and in another example, a system 200 is provided that includes an apparatus 204 and a separate camera 206. The system 200 may optionally include one or more first sensors 212 and/or second sensors 218.

Fig. 2 illustrates a potential implementation of a device 202 that includes an apparatus 204 and a camera 206. The device 202 may optionally further include one or more of the following additional components:

a display 208 (e.g., using light emitting diodes, liquid crystals, or other known underlying technologies);

a user interface 210 (e.g., a touch screen, buttons, sliders, or other known underlying technology);

a device 204;

at least one first sensor 212; and

an input/output device 214 configured to transmit and/or receive data between the device 202 and an input/output device 220 of an external device (e.g., a wearable or other device) using a wired or wireless connection.

The device 202 of fig. 2 may optionally include multiple cameras, such as one or more front-facing cameras and/or one or more rear-facing cameras.

The device 202 of fig. 2 may be a handheld portable electronic device 202. The handheld portable electronic device 202 may be a smart phone, a tablet computer, or a laptop computer. The handheld portable electronic device 202 may integrate all of the components of fig. 2 into one housing. The weight of the device 202 may be less than 500 grams to enable the device 202 to easily remain at arm length to capture "self-portrait" style images and videos.

Potential use cases for the methods described herein include performing these methods during a video communication session, where image data (e.g., from camera 206) is communicated between devices. The devices may be separated across a wide area network and/or a local area network. The video communication session may be managed by a software application configured for one or more of: video conferencing; video chat; video sharing; or a video stream. The communication may be unidirectional or bidirectional. The displayed indication/image data may be a real-time feed as described above and the methods 100, 500 may be repeated frequently as described above. Other potential use cases include monitoring usage, e.g., emotional state or fatigue of staff or healthy patients may be monitored even when they are out of the lens of the camera 206.

In some implementations, the privacy option for blocking display of the indication 406 may be accessible via the user interface 210. This would be helpful to the following user, for example: users do not want to be aware of their emotional state when they are out of the lens of the camera 206.

In a formal interview via video conferencing, some participants may wish to talk privately outside of the shot. A simple single-press off/on control would be effective to display on the display 208 simultaneously with the other captured images described herein to avoid delays in switching between privacy options. However, the privacy option control may take any other form, depending on implementation requirements.

Fig. 11A illustrates an example of the controller 1100. The controller 1100 may be implemented as controller circuitry. The controller 1100 may be implemented in hardware alone, with certain aspects in software including firmware alone, or as a combination of hardware and software (including firmware).

As illustrated in fig. 11A, the controller 1100 may be implemented using instructions that implement hardware functionality, for example, by using executable instructions of a computer program 1106 in a general-purpose or special-purpose processor 1102, which may be stored on a computer readable storage medium (disk, memory, etc.) for execution by such a processor 1102.

The processor 1102 is configured to read from and write to the memory 1104. The processor 1102 may also include an output interface via which the processor 1102 outputs data and/or commands and an input interface via which data and/or commands are input to the processor 1102.

The memory 1104 stores a computer program 1106 comprising computer program instructions (computer program code) which, when loaded into the processor 1102, control the operation of the apparatus 204. The computer program instructions of the computer program 1106 provide the logic and routines that enables the apparatus to perform the methods illustrated in fig. 1 and 5. The processor 1102 is able to load and execute the computer program 1106 by reading the memory 1104.

The apparatus 204 thus comprises:

at least one processor 1102; and

at least one memory 1104 including computer program code,

the at least one memory 1104 and the computer program code are configured to, with the at least one processor 1102, cause the apparatus 204 to perform at least:

receiving information indicative of a relative arrangement of the object and the camera 206; determining from the information whether a condition associated with the relative arrangement of the object and the camera 206 is satisfied; and if the condition is satisfied, causing an indication of at least one detected characteristic of the display object to be displayed 406.

As illustrated in fig. 11B, the computer program 1106 may arrive at the apparatus 204 via any suitable delivery mechanism 1106. The delivery mechanism 1106 may be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a storage device, a recording medium such as a compact disc read only memory (CD-ROM) or Digital Versatile Disc (DVD), or a solid state memory, an article of manufacture that includes or tangibly embodies the computer program 1106. The delivery mechanism may be a signal configured to reliably transfer the computer program 1106. The apparatus 204 may propagate or transmit the computer program 1106 as a computer data signal.

The computer program instructions are for causing an apparatus to perform at least the following or for performing at least the following: causing receipt of information indicative of a relative arrangement of the object and the camera 206; such that it is determined from the information whether a condition associated with the relative arrangement of the object and the camera 206 is satisfied; and if the condition is satisfied, causing an indication of at least one detected characteristic of the display object to be displayed 406.

The computer program instructions may be embodied in a computer program, a non-transitory computer readable medium, a computer program product, a machine readable medium. In some, but not necessarily all, examples, the computer program instructions may be distributed over more than one computer program.

Although memory 1104 is illustrated as a single component/circuitry, it may be implemented as one or more separate components/circuitry, some or all of which may be integrated/removable and/or may provide permanent/semi-permanent/dynamic/cached storage.

Although the processor 1102 is illustrated as a single component/circuitry, it may be implemented as one or more separate components/circuitry, some or all of which may be integrated/removable. The processor 1102 may be a single-core or multi-core processor.

References to "computer-readable storage medium", "computer program product", "tangibly embodied computer program", etc. or to a "controller", "computer", "processor", etc., should be understood to encompass not only computers having different architectures such as single/multi-processor architectures and sequential (von neumann)/parallel architectures, but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device, whether instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device etc.

As used in this application, the term "circuitry" may refer to one or more or all of the following:

(a) hardware-only circuitry implementations (such as implementations in analog-only and/or digital circuitry) and

(b) a combination of hardware circuitry and software, such as (as applicable):

(i) combinations of analog and/or digital hardware circuitry and software/firmware, and

(ii) a hardware processor (including a digital signal processor) having software, and any portion of memory that work together to cause a device such as a mobile phone or server to perform various functions, and (c) hardware circuitry and/or a processor, such as a microprocessor or a portion of a microprocessor, that requires software (e.g., firmware) for operation, but may not be present when software operation is not required.

This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also encompasses purely hardware circuitry or an implementation of a processor and its (or their) accompanying software and/or firmware. For example and if applicable to the particular claim element, the term circuitry also encompasses baseband integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

The blocks illustrated in fig. 1 and 5 may represent steps in a method and/or code segments in a computer program 1106. The description of a particular order of the blocks does not necessarily imply that a required or preferred order exists for the blocks but the order and arrangement of the blocks may be varied. Furthermore, it is also possible that some blocks are omitted.

Where a structural feature is described, it may be replaced by means for performing one or more of the structural feature's functions, whether such function or functions are explicitly or implicitly described.

The capture of data may include only temporary recording, or it may include permanent recording, or it may include both temporary and permanent recording. The temporary recording implies the temporality of the recording of the data. This may occur, for example, during sensing or image capture, at dynamic memory, at a buffer such as a circular buffer, register, cache, or the like. Permanent recording implies that the data is in the form of an addressable data structure that is retrievable from an addressable storage space and thus may be stored and retrieved until deleted or overwritten, but long term storage may or may not occur. The use of the term "capture" in relation to an image relates to the temporary or permanent recording of image data.

The systems, apparatus, methods, and computer programs may use machine learning, which may include statistical learning. Machine learning is a field of computer science that gives computers the ability to learn without explicit programming. If the performance of a computer's task in T (measured by P) improves with experience E, the computer learns from experience E that references some type of task T and performance measure P. The computer can typically learn from previous training data to make predictions for future data. Machine learning includes fully or partially supervised learning and fully or partially unsupervised learning. It can implement discrete outputs (e.g., classification, clustering) and continuous outputs (e.g., regression). Machine learning may be implemented, for example, using different approaches, such as cost function minimization, artificial neural networks, support vector machines, and bayesian networks. For example, cost function minimization can be used for linear and polynomial regression as well as K-means clustering. For example, an artificial neural network with one or more hidden layers models complex relationships between input vectors and output vectors. A support vector machine may be used for supervised learning. A bayesian network is a directed acyclic graph representing the conditional independence of a plurality of random variables.

The term "comprising" as used in this document has an inclusive rather than exclusive meaning. That is, any reference to X including Y indicates that X may include only one Y or may include more than one Y. If "comprising" is intended to be used in an exclusive sense, then in this context it will be clear by referring to "comprising only one.

In this description, various examples are cited. The description of features or functions relating to an example indicates that such features or functions are present in the example. The use of the terms "example" or "e.g.," or "can" or "may" in this text means that such features or functions are present in at least the described examples, whether described as examples or not, and that they may, but need not, be present in some or all of the other examples, whether or not explicitly described. Thus, "an example," e.g., "can" or "may" refer to a particular instance of a class of examples. The property of an instance may be only that of the instance or that of a class or that of a subclass of classes that includes some, but not all, instances in the class. Thus, implicitly disclosed are features described with reference to one example rather than another, which may be used as part of a working combination in other examples, when possible, but which do not necessarily have to be used in other examples.

Although embodiments have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the claims. The detected characteristics described in the above examples are facial expressions, which is an example of dynamic (time-varying) characteristics of expressions. In other examples, the characteristic is any other detectable dynamic characteristic of the expression. In a further example, the object may be any other object that possesses dynamic properties.

Features described in the foregoing description may be used in combinations other than the combinations explicitly described above.

Although functions have been described with reference to certain features, those functions may be performed by other features, whether described or not.

Although features have been described with reference to certain embodiments, those features may also be present in other embodiments whether described or not.

The terms "a" and "an" or "the" as used in this document have an inclusive rather than exclusive meaning. That is, any reference to X including one/the recited Y indicates that X may include only one Y or may include more than one Y unless the context clearly indicates the contrary. If the use of "a" or "the" is intended in an exclusive sense, it will be clear from the context. In some cases, the use of "at least one" or "one or more" may be used to emphasize an inclusive meaning, but the absence of such terms should not be taken to infer or an exclusive meaning.

The presence of a feature (or a combination of features) in a claim is a reference to that feature or to itself, and also to features that achieve substantially the same technical effect (equivalent features). Equivalent features include, for example, features that are variants and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

In this description, various examples have been described using adjectives or adjective phrases to describe example characteristics. Such description of a characteristic in relation to an example indicates that the characteristic is present in some examples entirely as described above, and is present in other examples substantially as described above.

Whilst endeavoring in the foregoing specification to draw attention to those features believed to be of importance it should be understood that the applicant may seek protection in respect of any patentable feature or combination of features hereinbefore referred to and/or shown in the drawings whether or not particular emphasis has been placed thereon.