阅读说明：本技术 对人-产品交互的无线测量 (Wireless measurement of human-product interaction ) 是由 史蒂芬·加里·布什 费兹·菲萨·谢尔曼 杰弗里·南泽 于 2020-03-03 设计创作，主要内容包括：本发明提供了推断产品活动,该推断产品活动包括：提供具有附接的第一谐波标签的第一产品；在该第一产品所在的第一区域处引导第一频率的第一发射信号；以及从该第一谐波标签接收第一返回频率的第一返回信号,其中该第一谐波标签在接收到该第一发射信号后辐射该第一返回信号,使得该第一返回频率为该第一频率的谐波。计算机然后可基于第一返回信号来推断使用第一产品的第一活动。(The invention provides inferred product activities, including: providing a first product having a first harmonic tag attached; directing a first transmit signal at a first frequency at a first region where the first product is located; and receiving a first return signal at a first return frequency from the first harmonic tag, wherein the first harmonic tag radiates the first return signal upon receiving the first transmit signal such that the first return frequency is a harmonic of the first frequency. The computer may then infer a first activity using the first product based on the first return signal.)

1. A method of inferring product activity, the method comprising:

providing a first product having a first harmonic tag attached;

directing a first transmit signal at a first frequency at a first region where the first product is located;

receiving a first return signal at a first return frequency from the first harmonic tag, wherein the first harmonic tag radiates the first return signal upon receiving the first transmit signal such that the first return frequency is a harmonic of the first frequency; and

inferring, by a computer, a first activity using the first product based on the first return signal.

2. The method of claim 1, the method comprising:

generating, by the computer, a power spectrogram based on a series of short-time Fourier transforms applied to the first return signal.

3. The method of claim 2, the method comprising:

analyzing, by the computer, the power profile with a convolutional neural network to determine identified activities,

wherein the identified activity comprises the first activity.

4. The method of claim 2, the method comprising:

analyzing, by the computer, the power spectrogram with a support vector machine to determine identified activities,

wherein the identified activity comprises the first activity.

5. The method of claim 2, wherein inferring the first activity comprises:

the power spectrogram signal is compared to one or more pre-stored activity models.

6. The method of claim 1, the method comprising:

directing a second transmit signal at a second frequency different from the first frequency at the first region;

receiving a second return signal at a second return frequency from a second harmonic tag, wherein the second harmonic tag is attached to a second product located in the first area, and the second harmonic tag radiates the second return signal upon receiving the second transmit signal such that the second return frequency is a harmonic of the second frequency; and

inferring, by the computer, a second activity using the second product based on the second return signal.

7. The method of claim 1, the method comprising:

receiving a second return signal at the first frequency from a user of a product in the first area that reflected the first transmitted signal.

8. A method of inferring cumulative usage of a product having an attached harmonic tag, the method comprising:

directing a transmit signal at a first frequency at a first region;

receiving a return signal at a second frequency from the harmonic tag, wherein the harmonic tag radiates the return signal upon receiving the transmit signal such that the second frequency is a harmonic of the first frequency;

determining, by a computer, one or more movement events of the harmonic tag over a period of time based on the return signal; and

inferring, by the computer, a cumulative usage of the product over the period of time based on the one or more movement events of the harmonic tag.

9. The method of claim 8, wherein inferring the cumulative usage over the period of time comprises:

accumulating, by the computer, a count of the one or more movement events occurring during the time period.

10. A system for inferring product activity, the system comprising:

a first product having a first harmonic tag attached;

a radar configured to direct a first transmitted signal of a first frequency at a first area in which the first product is located,

the radar is configured to receive a first return signal at a first return frequency from the first harmonic tag, wherein the first harmonic tag radiates the first return signal upon receiving the first transmit signal such that the first return frequency is a harmonic of the first frequency;

a memory storing executable instructions; and

a processor in communication with the memory, wherein execution of the executable instructions by the processor causes the processor to:

inferring a first activity using the first product based on the first return signal.

11. The system of claim 10, wherein execution of the executable instructions by the processor causes the processor to:

generating a power spectrogram based on a series of short-time Fourier transforms applied to the first return signal.

12. The system of claim 11, wherein execution of the executable instructions by the processor causes the processor to:

analyzing the power profile with a convolutional neural network to determine identified activities,

wherein the identified activity comprises the first activity.

13. The system of claim 11, wherein execution of the executable instructions by the processor causes the processor to:

analyzing the power spectrogram with a support vector machine to determine identified activities,

wherein the identified activity comprises the first activity.

14. The system of claim 10, further comprising:

the radar is configured to direct a second transmit signal at a second frequency different from the first frequency at the first region,

the radar is configured to receive a second return signal at a second return frequency from a second harmonic tag, wherein the second harmonic tag is attached to a second product located in the first area, and the second harmonic tag radiates the second return signal upon receiving the second transmit signal such that the second return frequency is a harmonic of the second frequency; and is

Wherein execution of the executable instructions by the processor causes the processor to:

inferring a second activity using the second product based on the second return signal.

15. The system of claim 10, further comprising:

the radar is configured to receive a second return signal of the first frequency from a user of a product in the first area that reflects the first transmitted signal.

Background

The present disclosure relates to tracking information about consumer products, and more particularly to tracking sports related information associated with use of consumer products.

There are several methods of Human Activity Recognition (HAR) based on spectrogram data (obtained via doppler effect, radar, sonar, etc.). Traditionally, a matching learning classifier for HAR is trained using manually selected features (e.g., low-level statistical parameters such as mean, variance, frequency, and amplitude) as input. Commonly used classifiers include Support Vector Machines (SVMs), decision trees, and Dynamic Time Warping (DTW). Such feature-based classifiers rely on domain knowledge and experience and often have drawbacks such as poor robustness and generality. More recently, one method of classifying HARs includes feeding raw amplitude spectrograms into a Deep Neural Network (DNN) so that the feature extraction step can be bypassed. Popular choices of DNN architecture include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Automatic Encoders (AEs). It has been shown that a hybrid model using various DNN structures (such as CNN or AE) as the combination of automatic feature extractor plus RNN as the classifier provides excellent performance.

Disclosure of Invention

One aspect of the invention relates to a method of inferring product activity, the method comprising: providing a first product having a first harmonic tag attached; directing a first transmit signal at a first frequency at a first region where the first product is located; and receiving a first return signal at a first return frequency from the first harmonic tag, wherein the first harmonic tag radiates the first return signal upon receiving the first transmit signal such that the first return frequency is a harmonic of the first frequency. The computer then infers a first activity using the first product based on the first return signal.

Another aspect of the invention relates to a method of inferring cumulative usage of a product having an attached harmonic tag, the method comprising: directing a transmit signal at a first frequency at a first region; and receiving a return signal at the second frequency from the harmonic tag, wherein the harmonic tag radiates the return signal upon receiving the transmit signal such that the second frequency is a harmonic of the first frequency. The computer may then determine one or more movement events of the harmonic tag over a period of time based on the return signal; and inferring a cumulative usage of the product over the time period based on one or more movement events of the harmonic tag.

Another aspect of the invention relates to a method of determining expiration of a defined useful life cycle for a product having an attached harmonic tag. The method comprises the following steps: storing, by a computer, a value indicative of a useful life cycle of a product; directing a transmit signal at a first frequency at a first region; and receiving a return signal at the second frequency from the harmonic tag, wherein the harmonic tag radiates the return signal upon receiving the transmit signal such that the return signal is a harmonic of the transmit signal. The computer may then determine, based on the return signal, one or more movement events of the harmonic tag within a time period starting from an earliest determined movement event in a current life cycle of the product; and accumulating a count of one or more movement events occurring during the time period. The computer may also determine whether the defined life cycle of the product has expired based on a count of one or more movement events that occurred during the time period.

Yet another aspect of the invention relates to a method of inferring movement, the method comprising: providing a first product having a first harmonic tag attached for human use; directing a first transmit signal at a first transmit frequency at a first region where the first product is located; receiving a first return signal at a first return frequency, wherein the first return frequency and the first transmit frequency are substantially the same; and receiving a second return signal at a second return frequency from the first harmonic tag, wherein the first harmonic tag radiates the second return signal upon receiving the first transmit signal such that the second return frequency is a harmonic of the first transmit frequency. The computer may then determine movement of the person based on the first return signal and determine movement of the first harmonic tag based on the second return signal.

One aspect of the invention relates to a system for inferring product activity, the system comprising: a first product having a first harmonic tag attached; a radar configured to direct a first transmit signal at a first frequency at a first area where the first product is located, the radar configured to receive a first return signal at the first return frequency from a first harmonic tag, wherein the first harmonic tag radiates the first return signal upon receiving the first transmit signal such that the first return frequency is a harmonic of the first frequency. The system further comprises: a memory storing executable instructions; and a processor in communication with the memory. In particular, execution of the executable instructions by the processor causes the processor to infer a first activity using the first product based on the first return signal.

Another aspect of the invention relates to a system for inferring cumulative usage of a product having an attached harmonic tag, the system comprising a radar configured to direct a transmit signal at a first frequency at a first area and to receive a return signal at a second frequency from the harmonic tag, wherein the harmonic tag radiates the return signal upon receiving the transmit signal such that the second frequency is a harmonic of the first frequency. The system further comprises: a memory storing executable instructions; and a processor in communication with the memory. In particular, execution of the executable instructions by the processor causes the processor to determine one or more movement events of the harmonic tag over a period of time based on the return signal; and inferring a cumulative usage of the product over the time period based on one or more movement events of the harmonic tag.

Another aspect of the invention relates to a system for determining expiration of a defined useful life cycle of a product having an attached harmonic tag. The system includes a radar configured to direct a transmit signal at a first frequency at a first region and to receive a return signal at a second frequency from a harmonic tag, wherein the harmonic tag radiates the return signal upon receiving the transmit signal such that the return signal is a harmonic of the transmit signal. The system further comprises: a memory storing executable instructions; and a processor in communication with the memory. In particular, execution of the executable instructions by the processor causes the processor to store a value indicative of a defined useful life cycle of the product; determining, based on the return signals, one or more movement events of the harmonic tag within a time period starting from an earliest determined movement event in a current lifecycle of the product; accumulating a count of one or more movement events occurring during the time period; and determining whether the defined life cycle of the product has expired based on the count of one or more movement events occurring during the time period.

Yet another aspect of the invention relates to a system for inferring movement, the system comprising: a first product having a first harmonic tag attached for human use; a radar configured to direct a first transmit signal at a first transmit frequency at a first area where the first product is located, the radar configured to receive a first return signal at the first return frequency, wherein the first return frequency and the first transmit frequency are substantially the same, and the radar configured to receive a second return signal at a second return frequency from a first harmonic tag, wherein the first harmonic tag radiates the second return signal upon receiving the first transmit signal such that the second return frequency is a harmonic of the first transmit frequency. The system also includes a memory storing executable instructions and a processor in communication with the memory. In particular, execution of the executable instructions by the processor causes the processor to determine movement of the person based on the first return signal; and determining movement of the first harmonic tag based on the second return signal.

Drawings

FIG. 1A illustrates an example environment of a product having harmonic tags according to the principles of the present disclosure.

FIG. 1B is an illustration providing more detail regarding the example environment of FIG. 1A.

Fig. 1C illustrates a radar and tagged object according to the principles of the present disclosure.

Fig. 1D illustrates an example antenna of a harmonic tag according to the principles of the present disclosure.

Fig. 2A-2C are flow diagrams of example methods of inferring information about harmonic tags according to principles of the present disclosure.

Fig. 3-4B illustrate example power spectrograms in accordance with the principles of the present disclosure.

Fig. 5-7 illustrate a process for utilizing different activity recognition models in accordance with the principles of the present disclosure.

Fig. 8 is a flow chart of an example process for determining movement of people and harmonic tags, according to the principles of the present disclosure.

Detailed Description

In the following detailed description of the illustrated embodiments, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration, and not of limitation, specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and that changes may be made without departing from the spirit and scope of the various embodiments of the present invention.

Embodiments in accordance with the principles of the present disclosure relate to monitoring the use of relatively low cost consumer products in at least some situations where radio or RFID tags cannot be embedded in the product due to cost and RF exposure issues. As explained below, useful information (e.g., usage data for replenishment) may be inferred if activities involving these types of products can be identified through non-video means. Examples of such products include toothbrushes, hair brushes, and containers, such as those containing laundry detergents, shampoos, toothpaste, and the like.

Generally, indoor radars have been used in research and commercial applications to analyze people and objects. For humans, a phenomenon known as the micro-doppler effect (very small frequency shifts due to the motion of objects reflecting radar signals) can be used to infer activities such as walking, running, falling, heartbeat, breathing, etc. However, it may be difficult to accurately measure and interpret signals from these activities, especially activities that may appear the same to a radar. The term "micromotion" is generally used to refer to the movement of a limb (e.g., leg, arm, hand) that moves relative to a larger object (e.g., a person's torso). The micro-motion features may be used to infer an activity that may be being performed by the monitored person.

Harmonic tags can be made small and typically consist of only a bent length of wire and a nonlinear electrical element (such as a diode). In some cases, harmonic tags have been used to detect presence. A harmonic tag is a tag that receives electromagnetic energy at one frequency (e.g., the fundamental frequency) but then retransmits the electromagnetic energy at a second frequency. Typically, the second frequency of the re-emitted or re-radiated energy is a harmonic of the fundamental frequency.

Since the harmonic tags re-transmit at different frequencies, their presence can be clearly distinguished from radar returns at fundamental frequencies, which can consist of reflections, constructive and destructive interference, etc. As described below, the harmonic tag may be associated with or attached to a particular object. Thus, the presence or motion of an object in the returned radar data from the tag can be used to refine the estimate of activity involving that object.

In addition, harmonic tags can be made to resonate at different frequencies. By transmitting at different frequencies and monitoring when different tags "appear" or "disappear," unique objects can be identified.

Referring now to the drawings and in particular to FIG. 1A, a general environment in which harmonic tagging and Doppler radar may be implemented is illustrated in accordance with the principles of the present disclosure.

A consumer 104 uses a product 102 in a first area 100 (such as a kitchen, laundry, bathroom, etc.). In the following description, the use of a toothbrush is taken as an example of an activity that a user may perform. Toothbrushes are provided by way of example only, and many different products are contemplated within the scope of the present invention.

The product 102 (e.g., toothbrush) may include a harmonic tag 103. Typically, the harmonic tag 103 is attached to the product 102 in an unobtrusive manner. The term "unobtrusive" is intended to mean that the label 103 does not interfere with or affect the normal use of the product 103. The harmonic tag 103 may be attached by the manufacturer before the product 102 is sold to the consumer, or the harmonic tag 103 may be a separate item that is attached to the product 102 after the product 102 has been obtained by the consumer 104.

The first area 100 may include other objects and furniture 106 that do not have harmonic tags attached or associated. As described below, the radar 108 is used to radiate energy at one frequency as a continuous wave or a pulsed wave, and the resulting return signal can be detected. The return signal may include a signal having a frequency of the transmit signal, and due to the harmonic tag, the return signal may include a signal having a frequency that is a harmonic of the transmit signal. Additionally, the radar 108 may radiate signals at multiple frequencies, for example, by sweeping a discrete frequency, thereby producing multiple return signals each at a different frequency.

The radar 108 may include a processor or computer 110 that processes the return signals and analyzes the return signals. It is also contemplated that the processor or computer 110 may be separate from the radar 108 but coupled to the radar 108 in order to receive signals from the radar 108. As explained below, analysis of the return signal can be used to infer an activity or movement event involving the product 102 with the associated harmonic tag 103.

FIG. 1B shows more detail regarding the radar 108 of the illustrated embodiment. In particular, radar 108 has a transmitter section 122 that radiates or transmits electromagnetic energy at one or more different fundamental frequencies. As shown in fig. 1B, four different fundamental frequencies 130A through 130D are shown by way of example. Thus, for purposes of example, four different harmonic tags 128A-128D are also shown. In radar 108, a separate transmitter may be included for each fundamental frequency, or alternatively, a single multi-frequency transmitter section sweeping through different fundamental frequencies may be used. Sweeping the frequency means transmitting each fundamental frequency individually for a predetermined period of time (e.g., 0.5 seconds or 2.0 seconds, etc.), and then repeating the transmission of the fundamental frequency signal.

The radar 108 has a receive section for receiving a return signal generated from the fundamental frequency signal. One receiver section 124 would likely be tuned to receive return signals 132A-132D at a fundamental frequency and caused by reflections of the transmitted fundamental frequency signals 130A-130D. The different receiver sections 126 will be tuned to receive return signals 134A-134D that are harmonics of the transmitted fundamental frequency signals 130A-130D. The presence of harmonic tags within the first region 100 will generate return harmonic signals that can be detected by the radar 108. If no harmonic tags are present in the first region 100, return signals at harmonic frequencies are not re-radiated to be detected by the radar 108. Each of the tags 128A-128D may be associated with one of the fundamental frequencies 130A-130D such that the tags 128A-128D are capable of generating respective return signals 134A-134D that are harmonics of the emitted fundamental frequency signals 130A-130D. From these return signals 134A-134D, radar 108 may detect the presence of different multiple products. Additionally, examples are provided below that use objects with attached harmonic tags to infer human activity. Using different harmonic tags for different objects allows for inferring the corresponding activity associated with the use of each different object. The different activities may occur simultaneously or may occur separately from each other at different times.

Fig. 1C shows details regarding an example radar 108A, in accordance with the principles of the present disclosure. Unlike the example of FIG. 1B, radar 108A of FIG. 1C is shown transmitting a single fundamental frequency. The transmitter 150 transmits a first signal 160 having a fundamental frequency in such a way that the antenna 122A directs the signal to a first area where the product 102 is located. The harmonic tag 103 is associated with the object 102, and the object 102 is in motion, as indicated by the "V" vector in fig. 1C. Thus, radar 108A detects two return signals. A return signal 162 is the reflected fundamental frequency signal and is detected by antenna 124A. In the example circuit shown, the return signal 162 may be filtered by the filter 156 to eliminate unwanted frequency components, such as frequencies above or below the fundamental frequency. In other words, the return signal may pass through a bandpass filter centered at the fundamental frequency. The filtered signal may then be mixed with the transmitted first baseband signal 160 in the mixer 154. Combining the two signals in this manner removes the fundamental frequency components, resulting in a baseband time domain signal centered at 0 Hz.

As is well known, movement of the person 104 and the object 102 causes the fundamental return signal 162 to include a component indicative of a slight doppler shift due to the motion. Although the motion of the object 102 and the person 104 may both contribute to the fundamental return signal 162, the movement of the person 104 contributes more to the fundamental return signal 162. Thus, analysis of the fundamental return signal 162 allows for a determination of how the person 104 is moving. The doppler shifted fundamental return signal 162 includes a component corresponding to the fundamental frequency and a component corresponding to the doppler shift caused by the motion of the person 104. The Doppler-shifted fundamental return signal has a frequency f_RSWhere f is the fundamental frequency in Hz, c is the speed of light in m/s, and v is the speed of the person in m/s. As described above, the doppler shifted fundamental return signal 162 is filtered by the filter 156 and then mixed with the transmitted first fundamental frequency signal 160 in the mixer 154. Combining the two signals in this manner removes the fundamental frequency components, again as described above, resulting in a baseband time domain signal centered at 0 Hz.

The baseband time domain signal may be processed using a well-known Short Time Fourier Transform (STFT) 166. In this way, a series of separate time slices (e.g., two seconds) of the fundamental return signal 162 can be processed and converted to a frequency domain signal. The result is a fundamental power spectrum 178 that includes frequency components and their corresponding amplitudes in the fundamental return signal 162 that result from the motion of the person 104 using the subject 102. Movement of person 104 toward antenna 158 produces a doppler shift in one direction (e.g., a positive sign), and movement of person 104 away from antenna 158 produces a doppler shift in a second direction (e.g., a negative sign). In the fundamental power spectrogram 178, the frequency values provide information about the speed of movement of the person 104, and the frequency amplitude values provide information about the "certainty" of the frequency values. For example, power spectrum 178 may indicate the presence of frequency components of about 20Hz with very high amplitudes and frequency components of about 10Hz with relatively low amplitudes. Automated analysis of this amplitude information may determine that the 20Hz frequency component is not due to noise, interference, or some other artifact of the radar's signal detection circuitry due to the higher amplitude. However, there may be some uncertainty with respect to the 10Hz frequency component because its amplitude is low, such that an automated analysis process may determine that the 10Hz frequency component may not actually be present in the fundamental power spectrum 177. As described above, the presence of frequency components in the fundamental power spectrogram indicates movement of a person, and more specifically, the velocity of the movement. The duration of movement (i.e., the integral of velocity over time) may provide a rough estimate of the amount or magnitude of movement of the person (e.g., 6 cm).

The radar of fig. 1C also includes an antenna 126A that receives a harmonic return signal 176 from the harmonic tag 103, the return signal 176 including a component indicative of a slight doppler shift due to motion of the object 102. Although a tag that provides a harmonic return signal, the tag is attached to the object 102. Thus, the harmonic return signal 176 is indicative of movement of the object 102. The doppler shifted harmonic return signal 176 includes components corresponding to harmonic frequencies and components corresponding to doppler shifts caused by motion of the object 102. The harmonic return signal 176 has a frequency f_HRSWhere n is the number of harmonics (e.g., 2), f is the fundamental frequency in Hz, c is the speed of light in m/s, and v is the speed of the person in m/s. The harmonic return signal 176 may also be filtered 170, such as with a band pass filter, and mixed with a signal that is a corresponding harmonic of the fundamental frequency. If the tag 103 re-radiates energy at the second harmonic, the second harmonic is generated using the frequency multiplier 164 and fed into the mixer 168. Similar to the above-described operation, the harmonic return signal 176 is converted to a baseband signal fluctuating at about 0Hz, which may be converted by a well-known short-time Fourier transform (STFT) 174 to produce a harmonic power spectrum 180 including harmonics due to motion of the object 102The frequency components in the wave return signal 176 and their corresponding amplitudes. Movement of the object 102 toward the antenna 172 produces a doppler shift in one direction (e.g., a positive sign), and movement of the object 102 away from the antenna 172 produces a doppler shift in a second direction (e.g., a negative sign).

Fig. 1D illustrates one example geometry of a harmonic tag 103 in accordance with the principles of the present disclosure. Specifically, fig. 1D illustrates a dual-band slot dipole antenna that may be constructed on a laminate substrate, such as a high frequency circuit material having a thickness of 1.52mm commercially available as RO3003 from Rogers Corporation (Rogers Corporation). The antenna may be constructed from a 17 μm thick copper laminate. The overall dimensions of the substrate may be 9.7cm by 2.8 cm. As shown in fig. 1D, the top horizontal leg may have a height of 2mm and a length of 19.5 mm. The vertical connecting section may have a length of 11mm and a height of 3.8 mm. The bottom horizontal leg may have a height of 1mm and a length of 31 mm. The bottom vertical element may have a height of 11.2mm and a length of 1 mm.

Thus, the example harmonic tag 103 includes an antenna 105 and a substrate 184 coupled to the antenna 105. The antenna 105 includes a top portion 180 and a bottom portion 182, respectively. The bottom portion 182 is designed to receive a transmitted signal (e.g., 2.5GHz), and the top portion 180 is designed to transmit or re-radiate a signal having a harmonic frequency of the transmitted signal (e.g., 5 GHz). In this example, the diode 184 is connected between two legs 182A and 182B that define the bottom portion 182 of the antenna 105. The antenna 105 may be constructed from a copper laminate as described above or from a conductive material that allows for the reception and transmission of electromagnetic energy, such as copper, nickel, tin, silver, aluminum, zinc, and/or alloys thereof. The substrate 184 enables the label 103 to be attached to a variety of objects conveniently and unobtrusively. As described above, the substrate 184 may be constructed of RO3003 material or may include polyester, polyimide, or similar material, and the antenna 105 may be coupled to the substrate 184 using an adhesive (such as an acrylic pressure sensitive adhesive).

As described above, one use of the radar and harmonic tags described with reference to fig. 1A-1C is to determine movement events and/or activities in which a consumer is using a product having an associated or attached harmonic tag. For example, with respect to the embodiment of fig. 1C, motion of person 104 is detected by determining doppler shift information in fundamental return signal 162 received by radar antenna 124A. Motion of the object 102 is detected by determining doppler shift information in the harmonic return signal 176 received by the radar antenna 126A. The return signal (either) may be divided into time segments (e.g., about 2 seconds), and each time segment will include doppler related information. Each time slice may be processed using the well-known STFT to produce a corresponding portion of the power spectrogram. STFT is a series of (possibly overlapping) fourier transforms and also a window function (e.g. a hamming window function) to reduce the start/end effect of pulling a finite section of the signal out of the series. The STFT provides a timeline of activities. The STFT will have to have a time slice that fits the activity. The term "suitable" depends on the activity that is detectable. For example, when a person is brushing their teeth, a pair of brush strokes (i.e., one stroke each in opposite directions) may occur every 1 second. A suitable time slice will be about 0.5 seconds to 1 second. A series of time slices each having that duration will allow multiple pairs of toothbrush strokes to be captured as separate movement events. Compared to STFT, a more conventional fourier transform will capture all frequency information, but may hide a single "event". The individual STFT time slices may be arranged in order to show how an object or person moves over a longer period of time (e.g., 5 to 45 seconds). As described above, the fundamental power spectrum relates to movement of the person 104, while the harmonic power spectrum relates to movement of the object 102. The doppler information includes information of the amount of motion experienced by the person or object. For example, the doppler shift frequency present in the return signal may correspond to the velocity at which the object (or person) is moving, the amplitude of the various frequencies present in the return signal may indicate the confidence or certainty that a particular frequency is actually present, and the periodicity (if any) of the return signal may indicate the time interval between successive occurrences of the particular frequency component. For example, when a consumer uses a toothbrush with an attached tag, a series of time slices of the harmonic return signal may indicate that the harmonic tag is moving back and forth at 2Hz, with a maximum speed of about 1 m/s. A single time segment may provide information about a single brush stroke or a pair of brush strokes, but may not reflect information about the periodicity of consecutive brush strokes or peak velocity in multiple brush strokes. However, the plurality of STFT time segments may be analyzed to determine peak velocities in the plurality of brush strokes and whether there is periodicity associated with any frequency components within a power spectrogram consisting of a series of STFT time segments. The sequence of time slices may be arranged to indicate that the back and forth motion is repeated every 2 seconds. This information, which can be extracted from the return signal, can be considered to define "characteristics" of the brushing. Whenever a return signal having similar characteristics to the brushing characteristics is captured at a later time, the computer may infer that human-related brushing activity is occurring.

By inferring human-related activities related to the product (i.e., product activities), then usage of the product may be determined. Brushing, combing, grooming, shaving, lifting containers, using paper towels, etc., are all examples of human-related activities involving products to which harmonic tags may be attached. This information may be used to determine when a product may need to be replenished or replaced.

Fig. 2A-2C are each an example flow diagram of a high-level view of using a harmonic tag in conjunction with a consumer product according to the principles of the present disclosure. Subsequent figures provide further details of the general steps listed in the flow diagrams of fig. 2A-2C.

Beginning at step 202 of fig. 2A, a first product having an attached harmonic tag is provided such that the product is for human use. In step 204, a transmit signal is directed at a first area where a product is located. The orientation of the emitted signal may be in a direction and height commensurate with how the product may be used, as it is desirable to detect movement of the object being used by the consumer. The transmitted signal has a first frequency which can conveniently be labelled as the fundamental frequency. The transmission signal may be a Continuous Wave (CW) signal that is always being transmitted or a pulsed wave signal that is periodically transmitted for a defined period of time (e.g., 0.2 seconds per 1.0 second or 0.02 seconds per 0.1 second). Alternatively, the emission signal may be provided by a system that also detects the presence of a person in the first area (e.g., passive infrared detection) before being powered on to emit the emission signal.

Harmonic tags such as that shown in fig. 1D are designed to receive a fundamental signal and radiate a return signal having a second frequency that is a harmonic of the fundamental frequency. Accordingly, in step 206, a return signal is received from the harmonic tag that radiates the return signal upon receiving the transmit signal. Steps 204 and 206 may be performed by conventional radar circuitry.

A processor, computer, or other type of processing device (such as a microprocessor, such as Sitara available from Texas Instruments)^TMOne of the families, or application processors, such as OMAP, also available from Texas Instruments^TMOne of the series, or a digital signal processor, such as the C6000 series, also available from Texas Instruments, or a microcontroller, such as STM32 available from STMicroelectronics^TMOne of a series) may be incorporated into the radar or may be a separate processor, computer or other processing device such that the radar provides a return signal to the separate processor, computer or other processing device. As discussed above, as the harmonic tag moves, the radiated harmonic return signal may be doppler shifted (positive and negative) rather than simply being a pure harmonic of the fundamental frequency. By eliminating the harmonic frequencies from the harmonic return signal, a baseband signal is generated that will vary over time as the tag is moving. The presence of a change in the signal indicates that the tag and object are moving. The return signal of a stationary tag will not include changes due to doppler shift to the radiated signal. As described above, a harmonic spectrum may be constructed from harmonic return signals using STFT. An automated process using a processor, computer, or other processing device may determine whether a harmonic tag (or an object to which it is attached) is moving by analyzing pixel values present in a harmonic spectrogram. If there is no movement of the object, the amplitude of any frequency components in the harmonic spectrum above or below 0Hz will be substantially zero. However, if there is movement of the object, one or more frequency components of the harmonic spectrogram will have non-zero amplitudes. One of ordinary skill will recognize that a predetermined threshold may be applied such that frequency components having amplitude values (i.e., pixel values in a harmonic spectrogram) below the predetermined threshold are still considered to be absent, even if the vibration is presentThe amplitude is not exactly 0. Noise, interference and other unintended artifacts of the receiving and processing circuitry may inadvertently cause frequency components of the harmonic spectrogram to have non-zero, but very small amplitude values, even if the frequency components are not actually caused by object movement. Similar analysis may be performed with respect to the pixel data of the fundamental spectrogram to determine or detect whether there is movement associated with a person.

As discussed above, the movement of the tag and object may be characterized, for example, by a harmonic spectrogram comprising a timeline of different frequencies in a baseband version of the harmonic return signal and their amplitudes. Multiple samples of an activity (e.g., brushing) and their corresponding spectra may be captured. Different samples may relate to multiple persons of different ages and body types. One or more of these spectra corresponding to the sample activity may be compared to the recently captured and generated harmonic spectra to see if the recently captured harmonic spectra are similar to one of the one or more spectra corresponding to the sample activity. Different spectra may be generated for a variety of different sample activities (such as shaving, brushing, etc.), and thus the most recently captured and generated spectra may be compared to the different spectra corresponding to the sample activities to determine a spectrum similar to the most recently captured spectra. Thus, in step 208, the computer or processor may infer activity that is using the product to which the harmonic tag is attached based on the harmonic return signal and the resulting harmonic spectrogram, which relate to information about movement of the harmonic tag.

Conventional image analysis techniques, such as, for example, cross-correlation, may be utilized with respect to comparing one spectrogram to another spectrogram or comparing a portion of a recently captured spectrogram to one or more other spectrograms corresponding to sample activity. For image processing applications where the brightness of the image and template may vary due to lighting and exposure conditions, the image may be first normalized. This is usually done in each step by subtracting the mean and dividing by the standard deviation. That is, the cross-correlation of the template t (x, y) with the subimage f (x, y) is

Where n is the number of pixels in t (x, y) and f (x, y), μ_tIs the average value of t (x, y), μ_fIs the average of f (x, y), σ_tIs the standard deviation of t (x, y), and σ_fIs the standard deviation of f (x, y). One of ordinary skill will readily recognize that the average μ in the above equation may also be subtracted without ambiguity_tAnd mu_fThe cross-correlation is calculated in the case of (1). Normalized correlation is one of the methods used for template matching, a process for finding the incidence of patterns or objects within an image. The template is moved to different locations of the spectrogram for known activities and cross-correlation values are calculated at each of the different locations. In accordance with the principles of the present disclosure, a current spectrogram, or a portion of a current spectrogram (fundamental or harmonic), can be considered a template to compare with a spectrogram of known activity. A cross-correlation score above a predetermined threshold indicates that image features of the current spectrogram, or a portion of the current spectrogram, are similar to features in a spectrogram of known activity. In this way, the activity of a person using a subject with an attached harmonic tag may be inferred from the return signals of the harmonic tags used to generate the current spectrogram.

The flow chart of fig. 2B depicts a method for inferring information about usage of a product with a harmonic tag attached. The meaning of "usage amount" will vary depending on the type of product. The amount of use of the razor may relate to how many times it is used, where each "use" corresponds to the razor moving from and returning to an original position, or may relate to how far the razor has traveled, i.e. the distance. The amount of toothpaste container used may relate to the frequency with which it moves from and returns to the original position. The amount of toothbrush used may relate to how many different brush strokes are detected. The amount used may also relate to a specific time period. For an air freshener, the relevant time period (and usage) can be a fixed time period (e.g., 3 months) after the cartridge is replaced. The relevant time period for the toothbrush may be open, but will start to be measured when the product is first used.

Similar to the flowchart of fig. 2A, the flowchart of fig. 2B begins at step 230, where the transmitted signal at the first or fundamental frequency is directed to a first area where a product with an attached harmonic tag is located. As previously mentioned, harmonic tags are designed to receive a fundamental frequency signal and radiate a harmonic return signal having a second frequency that is a harmonic of the fundamental frequency. Accordingly, in step 232, a harmonic return signal is received from a harmonic tag that radiates the harmonic return signal upon receiving the transmit signal. Steps 230 and 232 may be performed by conventional radar circuitry.

In step 234, the computer may determine one or more movement events of the harmonic tag based on the harmonic return signal and the doppler information contained therein. Similar to the discussion above regarding "usage," the term "movement event" may vary depending on the product to which the harmonic tag is attached. Screwing the closure onto or off of the container (i.e., the closure moving through a predetermined angle) may define a movement event. Determining that the container (e.g., for laundry detergent) has moved from an original position and/or returned to an original position may comprise one or more movement events for that type of product, i.e., movement of the container from an original position to another position, such as on or near a washing machine, may comprise one movement event, and movement of the container from on or near a washing machine back to an original position may comprise another movement event. It is also contemplated that the one or more movement events may comprise activities, for example wherein two movement events of the laundry detergent container may comprise a single activity of a single use of the laundry detergent container by a consumer. The movement event of the razor may be each of one or more different strokes in one or more directions. A movement event may be determined within a particular time period. Typically, this period will begin when the consumer first uses the product and will continue until the product is replaced or replenished. The first detection of the presence of a harmonic tag may be one way for a computer or processor to automatically determine that the time period should begin. Alternatively, a system is contemplated in which a consumer may use a user interface to indicate that a time period for detecting a movement event for a product should begin. For example, the graphical user interface screen may list products detected in the first area and allow the consumer to select the appropriate product and select to reset or begin the time period for which the movement event is to be detected.

Examples are provided herein that describe how activity can be inferred from harmonic return signals or a combination of harmonic return signals and fundamental return signals. With respect to "movement events," and as described above, an activity may include one or more movement events. As discussed above, in one example, the user's activity may be moving the detergent container between two positions. In this case, the first movement event may also be defined as moving the detergent container from the home position to another position, and the second movement event may be defined by moving the detergent container from another position back to the home position. Accordingly, the techniques and methods described herein in connection with inferring activity are equally applicable to inferring or determining movement events. In other cases, an activity may be defined as "brushing teeth". The activity may consist of more than one movement event. For example, a single movement event may be defined as a brush stroke in one direction, a single movement event may be defined as a pair of successively occurring brush strokes having one stroke in each direction, or two movement events may be defined as a pair of successively occurring brush strokes having one stroke in each direction. As described above, a power profile of a known activity may be used to compare to recently captured and generated power profiles to infer activity. In a similar manner, a known power spectrogram can provide finer granularity such that a power spectrogram associated with a corresponding known or defined movement event can be utilized. Accordingly, the techniques and methods described herein relating to inferring activity are equally applicable to inferring or determining movement events based on return signals (e.g., harmonic return signals).

Based on the determined movement event over a period of time corresponding to use of the product having the harmonic tag, in step 236, the computer may infer an accumulated usage of the product associated with the harmonic tag during the period of time. Such information may be displayed to the consumer (using the GUI described above), the information may be collected and sent to the wireless device, or the information may be collected and transmitted to a store, distributor, manufacturer, or other data collector, possibly to automate the purchase of restocked products where appropriate.

FIG. 2C is a flow chart of a number of conceptual methods that may be built upon the flow chart of FIG. 2B. First, in step 250, a value is stored in the computer that indicates or represents what is considered to be a defined useful life cycle for the product to which the harmonic tag is attached.

Similar to the flowchart of fig. 2A, the flowchart of fig. 2C includes a step 252 in which a transmission signal at a first or fundamental frequency is directed to a first area where a product with an attached harmonic tag is located. As previously mentioned, harmonic tags are designed to receive a fundamental frequency signal and radiate a return signal having a second frequency that is a harmonic of the fundamental frequency. Accordingly, in step 254, a harmonic return signal is received from a harmonic tag that radiates a harmonic return signal upon receiving the transmit signal. Steps 252 and 254 may be performed by conventional radar circuitry.

In step 256, the processor or computer may determine one or more movement events of the harmonic tag based on the harmonic return signal and the doppler information contained therein, similar to the discussion above with respect to fig. 2B. However, in step 256, the time period for which the movement event is determined is defined as the time period from the earliest determined movement event in the current life cycle of the product. In step 258, the number of movement events that occurred during the time period is represented by accumulating counts of different movement events that occurred during the time period. Finally, in step 260, the computer that accumulates the count and stores a value indicative of the defined useful life cycle may use both data elements to determine whether the defined useful life cycle of the product has expired.

FIG. 3 depicts a simplified schematic representation 302 of an amplitude spectrogram (such as the harmonic spectrogram 180 of FIG. 1C or the fundamental spectrogram 178 of FIG. 1C). Once fundamental return signal 162 or harmonic return signal 176 have been downconverted, either spectrogram 178, 180 will have a similar structure as shown in schematic representation 302.

The vertical axis 304 represents frequency in Hz and the horizontal axis 306 represents time. As described above, the return signal (fundamental or harmonic) may be filtered and down-converted to produce a time-domain baseband signal. A short portion of the time-domain baseband signal may be operated on by a short-time fourier transform to produce a series of frequency-domain samples arranged in the spectral representation 302 of fig. 3. Spectrogram signal 300 represents micro-doppler shift information in the return signal. In the bottom portion 310, the doppler shift frequency is negative, which corresponds to movement away from the transmitter. In the top portion 308, the doppler shift frequency is positive, which corresponds to movement toward the transmitter.

Fig. 4A and 4B depict actual movement-related spectrograms in accordance with the principles of the present invention. Although shown here in grayscale, the spectrogram is typically colored such that the color represents the amplitude associated with a particular frequency. Fig. 4A shows a fundamental frequency spectrum plotting information related to the movement of a person. FIG. 4B shows a harmonic spectrum plot of information related to movement of harmonic tags.

Further in accordance with the present disclosure, radar echo data may be used to classify activities (particularly human-related activities) into a limited number of classes through a process of spectrogram preprocessing and machine learning classification. Preprocessing of the spectrogram may include computing a short-time fourier transform of the time series radar signal. The pre-processing may also include a contrast enhancement step in which a filter is applied to amplify frequency features believed to be relevant to the classification and attenuate those frequency features believed to be attributable to noise. Contrast enhancement may be beneficial when pixel values of an image are clustered close to each other. As is known in the image processing art, an automated process may analyze a range of pixel values present in an image and increase contrast by spreading actual pixel values over the entire range of potential pixel values.

One type of machine learning classifier is a Support Vector Machine (SVM). With this type of classifier, the preprocessing of the spectrogram will include one or more feature extraction or identification steps in which the presence or periodicity of energy in the spectrogram is identified. One or more features may be extracted using rules provided by the model architecture. For example, the frequency fluctuation is divided into intervals (e.g., components), and the size of the energy in a particular interval relative to other intervals is looked at. Or an interval or frequency component is selected and observed over time to estimate periodicity. Another technique for automatically determining periodicity is to apply a two-dimensional fourier transform to some or all of the most recently collected spectrograms. The resulting transformed image will indicate the periodicity of one or more frequency components in the power spectrogram. In other words, applying the STFT to create the initial power spectrogram may indicate the presence of 20Hz frequency components in the return signal as indicated by the pixel values in the power spectrogram, while the 2D fourier transform of the power spectrogram may indicate the presence of 20Hz frequency components every 2 seconds.

For example, in a spectrogram of a doppler echo of a person walking, the torso may provide a small frequency shift as the person walks away from the radar, but the swinging arms and legs will provide oscillating traces that vary between negative to positive values of the frequency shift. Once all the extracted or identified features required for SVM model inference have been extracted, they can be fed to the SVM model as a vector for classification.

As one of ordinary skill will appreciate, SVM models are generated by a computer or similar processing device using known test data. For example, a large number (e.g., hundreds or thousands) of spectrogram spectra may be collected for different activities, and for each spectrogram, relevant features are identified. In the event that the activity associated with the spectrogram is known and relevant features of the spectrogram are identified, the machine learning algorithm may automatically construct the SVM model without additional human assistance.

FIG. 5 shows an overview of the process of using SVM to recognize and classify human activity. Baseband time domain signal data 502 is generated as described with reference to the previous figures. Next, a short-time fourier transform 504 may be applied to generate spectrogram data 506. The spectrum 506 may be a fundamental spectrum (such as 178 of FIG. 1C) or a harmonic spectrum (such as 180 of FIG. 1C).

Prior to the process shown in fig. 5, the subject matter expert has identified features that may be in the spectrogram that are relevant to classifying the spectrogram as being associated with a particular activity. One example of a feature may be that a particular frequency component (e.g., 20Hz) occurs with a period of about 1 second. Another feature may be the presence of different frequency components (e.g., -20Hz) between each pair of adjacent 20Hz components. In this way, a rule-based feature extraction step 508 may be performed. In other words, a set of analysis and calculation steps may be applied (automatically or manually) to evaluate the spectrogram 506 to determine the extent to which certain features are present in the spectrogram 506. Typically, the feature vector includes elements and corresponding values for each extracted feature. Some of the feature values may be binary values, such as a "1" if a feature is present, or a "0" if no feature is present. The feature value may relate to a percentage, such as "50% of the spectrogram pixels are green". The feature value may be an amplitude of each frequency component in the spectrogram. One example feature and corresponding value may be a standard deviation (or some other moment) of the frequency energy at a particular time slice. Another example feature and corresponding value may be the magnitude of a particular frequency value normalized by other frequency values. One of ordinary skill will recognize that many possible features and values may be determined by the subject matter expert such that rule-based feature extraction 508 may be performed.

Support vector machine model 514 is also generated prior to the other steps of fig. 5. As described above, SVM model 514 is generated by providing training data to a machine learning algorithm for automatically producing SVM model 514, which serves as activity recognition model 512 to evaluate feature vector 510 derived from features extracted via spectrogram 506. The result is that the activity recognition model classifies 516 the spectrogram as being associated with a particular activity. The activity recognition model 512 defining the SVM model 514 may be implemented on a computer or processing device having sufficient resources to generate the classification 516 within the time frame desired by the system designer. The time frame may vary depending on whether the system is a real-time system or not.

Other types of machine learning classifiers, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), may employ frequency data from the preprocessing step as input. In this case, the short time periods of the spectrogram are fed into a classifier model, which may include a convolution function that emphasizes or attenuates features based on training data, and a recursive function that infers activity from temporal correlations in the time series data, such as Long Short Term Memory (LSTM) units. The duration of time series data given to the model for classification will vary based on activity, but may range from 0.1s to 2 s. These models are trained using a large amount of labeled data, so that feature extraction and temporal relevance weights will be sufficiently generalized not just to how one or two people are doing the activity.

As one of ordinary skill will recognize, CNN models are generated by a computer or similar processing device using known test or training data. For example, a large number (e.g., hundreds or thousands) of training spectrograms may be collected for different activities. The process may then begin, where the machine learning algorithm applies a series of convolution kernels to the image of each spectrogram. The series may typically be randomly selected convolution kernels of different sizes and different weights in each cell of the kernel. Different kernels may tend to emphasize different graphical features of the spectrogram, such as edges, colors, object sizes, proximity of different objects. The result of the machine learning algorithm is to automatically identify one or more convolution kernels that can effectively classify the activity associated with the spectrogram. The generation of the CNN model is performed automatically by a computer or similar processing device, in addition to human assistance in collecting training spectrograms, labeling activities associated with each spectrogram, and providing the labeled training spectrograms to a machine learning algorithm.

As mentioned above, RNNs can identify time series of different spectra. In other words, a first spectrogram having a first set of features may be temporally followed by a second spectrogram having a second set of features. Thus, not only do the individual spectrograms provide information that helps classify an activity, but the sequence of spectrograms that are related to each other may also provide related information. The training of RNNs is similar to that of CNNs in that training data (i.e., spectrograms) is provided to a computer or similar processing device that automatically builds RNN models. When a non-training spectrogram of a user is collected according to embodiments of the present disclosure, the spectrogram may have a learned convolution kernel applied to a CNN model that extracts features from the spectrogram. The time-ordered sequence of these spectrograms may then be fed into an RNN model that infers user activity.

FIG. 6 shows a process overview of using CNNs and RNNs to identify and classify human activity. Baseband radar data 602 is generated as described with reference to the previous figures. Next, a short-time fourier transform 604 may be applied to generate spectrogram data 606. The spectrum 606 may be a fundamental spectrum (such as 178 of FIG. 1C) or a harmonic spectrum (such as 180 of FIG. 1C).

As described above, a variety of training data is collected relating to the activity for which recognition is desired. The training data includes a number of spectrograms, each spectrogram having been labeled as being associated with a particular activity. Further, each of the training data "elements" may comprise a chronologically ordered spectrogram sequence, rather than only a single spectrogram. The training data may first be used in a deep learning algorithm to automatically generate a multi-layer Convolutional Neural Network (CNN). CNNs are generated in such a way that they learn which convolution kernels extract (or identify) spectrogram features that appear to be effective in correctly classifying training data. For a time-ordered spectrogram sequence, the features from each spectrogram can be sequenced and used as training data for a Recurrent Neural Network (RNN).

The CNN610 and RNN 612 are used as an activity recognition model 608 to evaluate the spectrogram 606. Spectrogram 606 is considered an image that CNN610 may operate on to extract one or more features (i.e., detect their presence in the image). The process of fig. 6 is not necessarily a one-time calculation, but may include a series of individual spectrogram patterns 606 received by the active recognition model 608, with each spectrogram pattern processed by the CNN 610. The resulting series of feature extraction data may then be provided to the RNN 612 for classification 614 of the series of spectrograms.

More detailed information about Deep Learning algorithms (such as CNN and RNN) is provided in Li et al, "A Survey of Deep Learning-Based Human Activity Recognition in radius", Remote Sensing 2019,11,1068, the disclosure of which is incorporated herein by reference in its entirety. More detailed information on feature extraction for SVM Machine learning models can be found in Kim et al, "Human Activity Classification Based on Micro-Doppler Signatures Using a Support Vector Machine", IEEE transactions, Geosci, remote Sens, 2009,47,1328-1337, the disclosure of which is incorporated herein by reference in its entirety.

As shown in fig. 1C, fundamental return signal 162 and harmonic return signal 176 may both be received and processed by radar 108. To improve the accuracy of the classification of the sensed activity, it may be advantageous to analyze the fundamental radar echo and the harmonic radar echo simultaneously. Fig. 7 depicts a combined machine learning model, where echoes from two receivers can be passed to separate convolutional layers, where features can be extracted based on the learned data. The combined data may be passed to a recursive layer that performs activity recognition.

Thus, in fig. 7, fundamental baseband radar data 702 is operated on by STFT 166 to produce spectrogram 178. The spectrogram 178 is a fundamental spectrogram and captures doppler shift related information caused by movement of the person 104. CNN 716 is similar to CNN610 of fig. 6 in that training data is collected and labeled for provision to a machine learning algorithm that generates the convolutional layer of CNN 716. Each convolutional layer includes a convolution kernel that is discovered by a machine learning algorithm to identify or emphasize features in the training data that are effective for classifying activities. Thus, each convolution kernel may be applied to the fundamental spectrogram 178 to effectively extract or emphasize features of the fundamental spectrogram 178 that help classify the spectrogram 178 as corresponding to a particular activity. Features may be extracted or identified in the fundamental spectrogram 178 using the CNN 716, and these features may then be provided to the RNN 720.

In addition, the harmonic band radar data 704 is operated on by the STFT 174 to produce the spectrogram 180. The spectrogram 180 is a harmonic spectrogram and captures doppler shift related information caused by movement of the harmonic tag 103. CNN 718 is similar to CNN610 of fig. 6 in that training data is collected and labeled for provision to a machine learning algorithm that generates the convolutional layer of CNN 718. Each convolutional layer includes a convolution kernel that is discovered by a machine learning algorithm to identify or emphasize features in the training data that are effective for classifying activities. Accordingly, each convolution kernel may be applied to the harmonic spectrogram 180 to effectively extract or emphasize features of the harmonic spectrogram 180 that help classify the spectrogram 180 as corresponding to a particular activity. The CNN 718 may be used to extract or identify features in the harmonic spectrum 180, which may then be provided to the RNN 720.

The RNN 720 is similar to the RNN 612 in that training data is collected and provided to a machine learning algorithm to generate the RNN 720. However, in this case, the training data may include features extracted from a pair of spectrogram types (i.e., fundamental spectrogram and harmonic spectrogram). In some cases, the spectrogram can be associated with data for approximately simultaneous motion. In other cases, the extracted features may be from fundamental and harmonic spectral data occurring at different times. Finally, the RNN 720 performs classification 722 of the activity captured by the radar and analyzed by the activity recognition model 714.

FIG. 8 is a flow diagram of an example method or process that relies on both a fundamental spectrogram and a harmonic spectrogram to infer information about movements that a user may be engaged in. Steps 802 and 804 are similar to steps 202 and 204 of fig. 2A. A first area in which a product having a harmonic tag is located is illuminated with a transmit signal having a first frequency (e.g., a fundamental frequency). Next, in steps 806 and 808, a return signal is received as a result of the transmit signal. One return signal is the fundamental return signal associated with the movement of a person using the product with the harmonic tag. Movement may include moving legs and arms as is done while walking, or movement may include moving arms, hands, feet, and other body parts while using the product. The frequency of the fundamental return signal is substantially the same as the frequency of the transmit signal.

The other return signal is a harmonic return signal and is generated by a harmonic tag that radiates the harmonic return signal upon receiving the transmit signal. The frequency of the harmonic return signal is a harmonic of the fundamental frequency. In step 810, the computer determines movement of the person using the product based on the fundamental return signal. As explained above, CNNs can be used to extract or identify features in a spectrogram that indicate a particular type of movement of a person and their body. The presence of movement can be determined by detecting the micro-doppler shift frequency in the fundamental spectrogram.

The movement of the harmonic tag may also be determined by the computer in step 812. As explained above, CNNs may be used to extract or identify features in a spectrogram that indicate a particular type of movement of a harmonic tag. The presence of movement may be determined by detecting the micro-doppler shift frequency in the harmonic spectrum.

The dimensions and values disclosed herein are not to be understood as being strictly limited to the exact numerical values recited. Rather, unless otherwise specified, each such dimension is intended to mean both the recited value and a functionally equivalent range surrounding that value. For example, a dimension disclosed as "40 mm" is intended to mean "about 40 mm".

Each document cited herein, including any cross referenced or related patent or patent application and any patent application or patent to which this application claims priority or its benefits, is hereby incorporated by reference in its entirety unless expressly excluded or otherwise limited. The citation of any document is not an admission that it is prior art with any disclosure of the invention or the claims herein or that it alone, or in combination with any one or more of the references, teaches, suggests or discloses any such invention. Further, to the extent that any meaning or definition of a term in this document conflicts with any meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to that term in this document shall govern.

While particular embodiments of the present invention have been illustrated and described, it would be obvious to those skilled in the art that various other changes and modifications can be made without departing from the spirit and scope of the invention. It is therefore intended to cover in the appended claims all such changes and modifications that are within the scope of this invention.