阅读说明：本技术 增强现实和虚拟现实操作系统 (Augmented reality and virtual reality operating system ) 是由 C.赖 A.席埃拉 于 2017-02-10 设计创作，主要内容包括：虚拟现实(VR)装置与投影显示平台通信,以建立与数据交换相关的安全通信会话。可以使用在VR显示器上接收的由投影显示平台呈现的QR码建立会话。VR装置捕获与待处理交易相关的语音或手势,并将适当的指令传递到投影显示平台以处理待处理交易。(A Virtual Reality (VR) device communicates with the projection display platform to establish a secure communication session related to the exchange of data. The session may be established using a QR code received on the VR display presented by the projection display platform. The VR device captures voice or gestures related to the pending transaction and passes appropriate instructions to the projection display platform to process the pending transaction.)

1. A system for gesture-based or voice-based approval, comprising:

an augmented reality/virtual reality (AR/VR) device worn by a user, the VR device comprising:

A first processor;

a first wireless communication device that transmits and receives signals with a projection display platform, the first wireless communication device broadcasting a signal with an identifier indicating AR/VR transaction capabilities while in proximity to the projection display platform;

A virtual reality projector associated with the display platform, the virtual reality projector projecting a virtual image in a field of view of a user;

A first cryptographic processor supporting mutual authentication with the projection display platform and encrypting transaction data;

one or more sensors to detect user gestures, facial expressions, iris focus, or voice commands interacting with the virtual image; and

A memory storing executable code and configuration data, the executable code for use by the first processor to:

Identifying the projection display platform;

Projecting an Augmented Reality (AR) image corresponding to transaction information received from the display platform;

in response to projecting the AR image, identifying an approval gesture or voice command indicating approval of a transaction described by the transaction information; and

In response to the approval gesture or voice command, sending transaction information and a transaction related password to the projection display platform; and

The projection display platform comprises:

A second processor;

A second wireless communication device that broadcasts a beacon signal indicating AR/VR transaction capabilities and provides bidirectional communication with a first wireless communication device of the VR device;

an input device for capturing transaction information;

A second cryptographic processor that supports mutual authentication with the AR/VR device and encrypts transaction data; and

A memory storing executable code and data, wherein the executable code causes the second processor to:

receiving the transaction information through the input device;

Identifying Augmented Reality (AR) capabilities in the VR device when the VR device is proximate to the projection display platform;

sending the transaction information to the VR device; and

Receiving transaction information from the VR device for completing a transaction.

2. The system of claim 1, wherein the first cryptographic processor encrypts the transaction information and the password prior to sending the transaction information to the projection display platform.

3. The system of claim 1, wherein the executable code causes the projection display platform to send a graphics file to the VR device, the graphics file including an image associated with the transaction information.

4. the system of claim 3, wherein the graphic file further comprises an image for an accepted payment brand.

5. the system of claim 3, wherein the VR device performs authentication of a user of the VR device with one of the one or more sensors prior to sending the transaction information to the projection display platform.

6. the system of claim 5, wherein the authentication of the user is one of a voiceprint or an iris scan.

7. the system of claim 1, wherein the one or more sensors comprise a motion detector.

8. the system of claim 1, wherein the one or more sensors comprise a camera.

9. the system of claim 1, wherein the first wireless communication device is a radio frequency device supporting one of a WiFi protocol and a bluetooth protocol.

10. the system of claim 1, wherein the first wireless communication device is an infrared device.

11. The system of claim 1, wherein the memory further stores the transaction information.

12. the system of claim 1, wherein the projection display platform presents a Quick Response (QR) code scanned by the VR device, the QR code returned to the projection display platform by the VR device as the identifier.

13. a method of communicating between a Virtual Reality (VR) device and a projection display platform, the method comprising:

Storing, at the projection display platform, transaction information corresponding to a transaction, the transaction information collected at the projection display platform;

Moving the VR device within proximity of the projection display platform;

Broadcasting a signal from the VR device indicating wireless transaction capabilities;

receiving the signal from the VR device at the projection display platform;

Activating a transaction approval protocol, the protocol comprising:

transmitting the transaction information from the projection display platform to the VR device;

presenting, by the VR device, the transaction information;

Receiving, by a sensor, an indicator at the VR device corresponding to instructions related to processing the transaction information;

authenticating, at the VR device, by capturing gestures of the user by a sensor array; and

sending the instructions from the VR device to the projection display platform; and

Processing the transaction at the projection display platform according to instructions received from the VR device.

14. the method of claim 13, further comprising: performing mutual authentication between the VR device and the projection display platform.

15. The method of claim 14, wherein transmitting the transaction information from the projection display platform to the VR device includes transmitting the transaction information over an encrypted channel using a session key formed during the mutual authentication.

16. The method of claim 14, further comprising scanning, at the VR device, a Quick Response (QR) code presented by the projection display platform, the QR code being part of a mutual authentication between the VR device and the projection display platform.

17. The method of claim 13, further comprising:

Presenting, at the VR device, images corresponding to one or more payment instruments; and

receiving, by the sensor, a second physical movement corresponding to selection of one of the one or more payment instruments.

18. The method of claim 13, wherein the projection display platform sends an image file to the VR device, the image file modified with the transaction information, the image file for use by the VR device in presenting the transaction information.

19. A method of communicating between a Virtual Reality (VR) device and a projection display platform, the method comprising:

storing, at the projection display platform, transaction information corresponding to a transaction, the transaction information collected at the projection display platform;

Displaying, at the projection display platform, a Quick Response (QR) code corresponding to the transaction information;

Capturing the QR code at the VR device and extracting transaction data from the QR code;

activating a transaction approval protocol, the protocol comprising:

Establishing an encrypted session between the VR device and the projection display platform using, at least in part, information from the QR code;

Presenting, by the VR device, the transaction information;

Receiving, by a sensor, instructions at the VR device related to processing the transaction information; and

sending the instructions from the VR device to the projection display platform through the encrypted session; and

processing the transaction at the projection display platform according to instructions received from the VR device.

20. the method of claim 19, further comprising authenticating at the VR device by capturing gestures of a user by a sensor array.

background

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

the hardware that processes the transaction requires the physical presentation of the payment instrument or tokenization device at the point of sale. For some people, physical interaction with hardware may be difficult or impossible. In some cases, there may be situations where there may not be any form of physical interaction between the entities involved in the transaction due to the deficiencies of the people involved in the transaction or the situations where they may find themselves in.

disclosure of Invention

The features and advantages described in this summary and the following detailed description are not all-inclusive. Many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. In addition, other embodiments may omit one or more (or all) of the features and advantages described in this summary.

The AR/VR device exchanges handshake data with the projection display platform via a wireless connection. The handshake data confirms hardware compatibility between the device and the platform in order to complete the approval operation by the AR/VR device. After establishing compatibility, the platform may send a transaction, in some embodiments, virtual reality image data, for presentation to a user of the AR/VR device as a projected VR image. The user may interact with the projected VR image to approve or reject the payment, or select a payment instrument. If approved, the AR/VR device may encrypt the response sent to the projection display platform and generate a password for the tokenized transaction. Interacting with the projected VR image avoids the need for the user to physically interact with the point of sale device, such as approaching, presenting a payment card, providing a signature, or co-locating a smartphone with a terminal.

drawings

FIG. 1 is a block diagram of a projection display platform in communication with an AR/VR device in accordance with the present disclosure;

FIG. 2 is a block diagram illustrating a system incorporating the projection display platform and AR/VR device of FIG. 1; and

FIG. 3 is a flow chart of an exemplary method of operating the system of FIG. 2.

The drawings depict preferred embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

Detailed Description

The projection display platform and AR/VR device are configured for automatic discovery and protocol negotiation so that a user can view and approve or reject a transaction without having to bring a payment card or smartphone into contact or close physical proximity with a point of sale device. Instead, the projection display platform and the AR/VR device communicate transaction details, payment instrument selections, approvals, and when available, tokenized card numbers and transaction related passwords over a wireless connection so that the user need only interact with the AR/VR device to complete the transaction.

FIG. 1 is a block diagram generally illustrating a representative AR/VR device 104 and a representative projection display platform 106. Projection display platform 106 may include a Central Processing Unit (CPU)108 operatively coupled to a wireless communication device 110, an input device 112, a cryptographic processor 114, and a memory 116.

The wireless communication device 110 may send and receive signals with the AR/VR device 104. In various embodiments, wireless communication device 110 may be a radio frequency device, such as an 802.11(WiFi) device or a Bluetooth device. In other embodiments, the wireless communication device 110 may be an infrared device or other optical frequency-based device. Wireless communication device 110 may support two functions and others. The first function may be to broadcast a beacon signal indicating that the projection display platform 106 supports AR/VR transaction capabilities. The second function may be to communicate data with the AR/VR device 104 during transaction processing. Of course, the wireless communication device 110 may support other functions, such as health and diagnostic reporting or other business and maintenance functions of the base unit.

the beacon signal may be an encoded message on a normal wireless channel, such as a set of characters appended to a Service Set Identifier (SSID) in a WiFi or bluetooth service discovery protocol (SPD). In other embodiments, the capability to support AR/VR may be broadcast during a protocol handshake or polling message. While it is not required that the beacon signal must be broadcast from the projection display platform 106, as opposed to from the AR/VR device 104, it is expected that for battery life reasons, it may be less common for the AR/VR device 104 to broadcast the beacon signal. In the case where the AR/VR device 104 may broadcast a beacon signal, the following process basically begins with a handshake process initiated by the opposite party. The projection display platform 106 may also include an input device 112 that may be used to capture transaction details. In an illustrative embodiment, the input device may be a barcode scanner, while in another embodiment, the input device may simply be a keyboard or touch screen for entering text and data.

The encryption processor 114 may be used for authentication and encryption related to transaction security. Especially in the case where the AR/VR device 104 has an electronic wallet, the mutual authentication may be based on a key issued by the wallet provider or an underlying payment card issuer encryption key. The crypto processor 114 may also be used to store master keys and develop derivative keys that are used to generate transaction-oriented passwords for those instances in the AR/VR device 104 that do not generate such a password.

The memory 116 may be any of numerous known physical memory devices and may include a combination of volatile and non-volatile memory, but does not include carrier wave or propagation media type memory. The memory 116 may include data and executable code stored in a plurality of modules, which may include an AR data formatting module 118, a transaction processing module 120, a communication/security protocol module 122, and an AR/VR image file 124. The AR data formatting module 118 may include data and executable code that receives transaction data from the input device 112 and extracts and/or reformats the data for use by the AR/VR device 104. For example, upon reading a Universal Product Code (UPC) for a particular product, the AR data formatting module 118 may look up information associated with the UPC, such as price, image, extended warranty data, associated discount offers, and the like. For color, depth, or stereo effects, text and image information may be formatted for presentation at the AR/VR device 104.

The transaction processing module 120 may have executable code that supports message formatting and communication with a processor 130, such as a merchant acquirer or issuer. The transaction processing module 120 may also manage the message format and content passed between the projection display platform 106 and the AR/VR device 104. The communication and security protocol module 122 may manage various externally interactive communication protocols, and may also include security protocol management in conjunction with the cryptographic processor 114.

The AR/VR image files 124 may include previously prepared image files for use by the AR/VR device 104 for presentation to the user 102. The images stored in the AR/VR image file may also be generated in real-time during the interaction. For example, information such as a product image may be retrieved, and a stereoscopic image may then be generated and stored for transfer to and use by the AR/VR device 104.

Turning to the AR/VR device 104, the AR/VR device 104 may be a stand-alone device carried or worn by the user 102. In other embodiments, the AR/VR device 104 may be part of another personal device. For example, a personal electric scooter or powered wheelchair may include AR/VR capabilities that support interaction in various embodiments of the projection display platform 106 (e.g., point-of-sale devices) using the described techniques.

the AR/VR device 104 may include a CPU132 that executes instructions stored in a memory 144. The AR/VR device 104 may also include a wireless communication device 134 configured to communicate with the wireless communication device 110 of the projection display platform 106. The AR/VR projector 136 may project an image to the field of view of the user 102.

Augmented Reality (AR) may generally be understood as the ability to project information, such as images or text, over a user's direct viewing area. Virtual Reality (VR) can generally be understood as creating a visual environment that replaces the user's direct viewing area with a captured image projected in the user's field of view. In practice, this scenario may be the actual environment in which the user 102 interacts with the additional information available. For purposes of this disclosure, it is immaterial whether the AR/VR device 104 is an augmented reality device, such as a Google Glass or a virtual reality device (e.g., an Oculus Rift). The distinction between AR and VR is not important as long as the user 102 is able to feel its actual location and is able to interact with the surrounding physical environment.

The encryption processor 138 may be used to generate derivative keys, sign/verify, encrypt and decrypt various data items as part of mutual authentication with the projection display platform 106, data security to interact with the downstream processor 130 (if any), passwords to generate transaction packets, and the like. One or more sensors 140 may be used to detect the area around the user 102, in particular to capture images of the immediate environment, as well as to detect movement in three-dimensional (3D) space in the user's field of view. Detecting movement may be used to make a selection and indicate approval or denial of an offer, as discussed more below, as well as to initiate payment and authenticate a user to a transaction approval platform.

the sensor array 140 may also have the ability to track the user's facial expressions as well as iris movement and focus as a component to initiate selection, payment and/or authentication processes.

a microphone 142 may be used as one of the sensors 140, but is shown separately to emphasize that the audio sensor and the spatial sensor may be used to gather information from the environment or from the user 102. Memory 144 may include multiple modules of executable code, data, or both. These modules may include a communication and security protocol module 146 that may perform similar functions as the corresponding communication/security protocol module 122 in projection display platform 106. The AR/VR projection module 148 may collect image data from the sensors, from local storage, from the AR/VR image file 124 of the projection display platform 106, and manage the required format and, if any, replacement of image data in the user's field of view.

the card and token data 150 may be a user wallet or local repository of other information such as a Personal Account Number (PAN) for financial instruments that the user 102 may use to complete transactions, reward account registration, etc. The gesture recognition module 152 may interpret signals received via the sensor 140 or the microphone 142 as instructions from the user 102. This sensing process may be activated after the transaction has reached a point in time at which a response is expected to be received from the user 102, e.g., after an AR/VR image has been projected into the user's field of view. These gestures may include nodding, waving a hand in the field of view, touching a hand or tool to the projected image, or recognizing a predetermined audio signal.

the memory 144 may also include cryptographic questions 154, such as a master key, derivative key, certificate, password, or similar data used in completing an authentication function or transaction. In one embodiment, the cryptographic questions 154 may be located in a secure area of the memory 144, such as a secure element, or may themselves be locally encrypted by the cryptographic processor 138.

The authentication and transaction security module 156 may include executable instructions and policy data for identifying, enforcing, and formulating security protocols associated with particular phases of interaction with the projection display platform 106 or downstream processor 130, such as a merchant acquirer.

an overview of the system environment of the AR/VR device 104 and the projection display platform 106 is shown in fig. 2. The user 102 is associated with an AR/VR device 104. As described above, the user 102 may wear or carry the AR/VR device 104 to allow the AR/VR device 104 to project images into the user's line of sight, which may be done by glasses, headphones, or more advanced techniques (such as 3D projection). In embodiments where the user 102 uses a power seat or similar device, the projection may be projected onto a transparent or translucent screen. The AR/VR device 104 supports projecting images in the user's field of view and capturing input by the user 102 through gestures or voice commands.

The projection display platform 106 may wirelessly communicate with the AR/VR device 104 via respective wireless communication devices 110, 134. The projection display platform 106 may broadcast a beacon signal indicating that AR/VR operational support is available so that the AR/VR device 104 may connect while in range. Projection display platform 106 may also communicate via a network 160, a private network such as a banking network or a public network such as the internet. Processor 130 may be a merchant acquirer, an issuer, a token service provider, or another entity involved in the transaction processing. The processor 130 may be an endpoint or may simply be an intermediary throughout the transaction approval process. The processor 130 may store stored information about the user 102 in the form of user data 102 a. Processor 130 may also include user data 162 for a plurality of other users, which data is explicitly communicated thereto or obtained over a period of time by continuously interacting with projection display platform 106 and other similar platforms (not depicted).

FIG. 3 is a flow diagram 300 of exemplary interactions between the projection display platform 106 and the AR/VR device 104. In this jump diagram, the activities performed by the projection display platform 106 are shown on the right side of the flowchart, while the activities performed by the AR/VR device 104 are shown on the left side of the flowchart. In the illustrated embodiment, at block 301, projection display platform 106 broadcasts a beacon signal. In some embodiments, the beacon signal may be continuously broadcast, while in other embodiments, the beacon signal may be transmitted at intervals. In other embodiments, the beacon signal may be broadcast only when the user 102 is sensed or by action of an individual, such as a store clerk, at the time the user 102 is present. As described above, other methods of discovering AR/VR interaction capabilities may be used, such as the AR/VR device 104 sending a request signal.

In the illustrated embodiment, the AR/VR device 104 receives the beacon signal at block 302 and responds with a signal confirming the AR/VR capabilities at block 304. Protocol negotiations may be maintained between the two parties to determine capabilities and formats and perform mutual authentication using any of a number of known protocols, such as ISO 14443. Such protocol negotiation and/or mutual authentication may be performed initially or at some other point in the process.

After mutual authentication, or in some cases without such mutual authentication, a session key may be derived at both the AR/VR device 104 and the projection display platform 106. These session keys may be used to encrypt data sent wirelessly between two machines to prevent eavesdropping, reducing the risk of man-in-the-middle attacks when used with key-based mutual authentication.

At block 305, for example, the projection display platform 106 may capture the transaction details using the input device 112. In one embodiment, the transaction details may include scanning one or more product barcodes or receiving a service request. These transaction details may be sent to the AR/VR device 104 and received by the AR/VR device 104 at block 306. In another embodiment, the projection display platform 106 may display a Quick Response (QR) code observed at the AR/VR device 104 by one of the sensors 140 so that the transaction details may be decoded from the QR code.

In an embodiment, the projection display platform 106 may send an AR/VR image file at block 307, which the AR/VR device 104 receives at block 308. As described above, the AR/VR image file may be related to the transaction being processed, with the AR/VR image file having an image related to the product or service purchased or having a payment instrument graphic extracted from a wallet account or another online resource identified during the above-described mutual authentication. In one embodiment, for long term storage of potentially large 3D image files, the AR/VR device 104 may have limited memory so that even what might be considered a generic image, such as brand logos representing different payment instruments, may not be stored on the AR/VR device 104 conventionally, but may be downloaded as needed at the time of the transaction.

At block 310, one or more AR/VR images may be projected into the user's line of sight and may include images of products, transaction details (such as a purchase total), or one or more payment instruments for approval and selection by the user 102. At block 312, the user 102 may indicate the selections with gestures or spoken commands. For example, in one embodiment, the user 102 may agree to a purchase total by saying "agree to pay" or a similar word or phrase after reviewing the projected image of the transaction details. In this embodiment, the user 102 may then be asked to select a payment instrument from a plurality of payment instruments. For example, a payment image may be displayed in VR space, and the user may use gestures, head movements, or language commands to move and select among the available options. In one embodiment, the payment instrument may be presented in a "rotating" carousel to allow viewing and selection. The user 102 may perform the selection by simply "touching" the projected image corresponding to the selected payment instrument. In various embodiments, for example, when transaction details are visible at the projection display platform 106, the transaction may be approved simply by touching or "swiping" the payment card flag. By monitoring motion in the field of view, particularly in the area (or frustum) covered by the projected image, the sensors 140 of the AR/VR device 104 can identify virtual "touches" or other gestures. The transaction receipt may be printed and contained in the purchased item or may be electronically sent to a wallet account or email address.

The gesture used to approve the transaction may then be verified against a predetermined gesture template, which may be configured during initial enrollment of the AR/VR device 104. That is, in one embodiment, the transaction approval may be a complex gesture that may be verified by the sensor array 140 as further authentication, rather than a simple gesture, such as pointing to a payment instrument. In another embodiment, authentication may involve an iris scan or other biometric indicator. Identification and verification of gestures or biometrics may be performed by the authentication and transaction security module 156 as an additional layer of security for the transaction.

If the transaction is not approved, the "no" branch may proceed from block 314 to block 315, and the transaction may be terminated. If the transaction is approved, the "yes" branch may proceed from block 314 to block 316, and in block 316, in one embodiment, a password may be generated for completing the transaction. A password that is the same as or similar to the password used for smart card payment may be used in conjunction with the tokenized card number to reduce the risk of card fraud. In some embodiments, the password may also incorporate the authentication information obtained above. In other embodiments, a password may not be generated at the AR/VR device 104 and a simple payment instrument value may be used to complete the transaction. At block 320, the transaction-related information captured or generated at the AR/VR device 104 may be sent to the projection display platform 106.

At block 321, approval information, such as passwords and tokens, may be passed to the projection display platform 106. In one embodiment, at block 323, additional details may be added to the information received from the AR/VR device 104, and the data packet may be sent to the processor 130 to complete the transaction according to the current policy. For example, the transaction data may be passed to a token service provider, where the token may be replaced by the issuer PAN, and the transaction passed to the appropriate issuer for ultimate approval of the transaction. Such approval may be communicated back to the terminal to allow the transaction to be completed. In one embodiment, approval details similar to those that may appear on a print receipt may be transmitted to and stored at the AR/VR device 104.

Fig. 4 illustrates an exemplary scene viewed by the user 102 at the projection display platform 106, which in this embodiment is shown as a point of sale (POS) device 402. In this example, the product is scanned by the barcode reader 404 and the aggregate of the purchase transactions may be displayed on the screen 406 of the POS device 402.

at this point, the projection display platform 106 may send the transaction details and VR images to the AR/VR device 104 over a wireless connection, as discussed above. In an alternative embodiment, the screen 406 may display a QR code with transaction details so that the AR/VR device 104 may capture the transaction details through a camera included as one of the sensors 140.

The image 408 may be projected into the user's field of view by the AR/VR device 104, in this case, the logo of the payment instrument and instructions to the user 102 indicating how to proceed. When the user 102 gestures on the projected image 408, as illustrated by act 410, the sensor 140 of the AR/VR device 104 may associate the area and direction of action (in this case by the user's hand) with the location of the projected image in order to infer approval of the transaction. Processing may continue as described above.

the above-described apparatus and methods overcome the technical challenges of improving accessibility at projection display platform 106 in one embodiment, such that user 102 does not need to approach or interact with projection display platform 106 by manually swiping, inserting, or tapping a payment instrument into projection display platform 106 or by entering a signature. The transaction elements required for secure and reliable processing are maintained while reducing or eliminating the need for direct access between the payment instrument and the projection display platform 106.

The disclosed apparatus and methods benefit merchants by extending many ways to serve customers in a secure manner. The user 102 has the added option of using the AR/VR device 104 to enhance the shopping experience and thus also benefits from being able to not only provide real-time recommendations when selecting products, but also reduce the physical requirements at the projection display platform 106 when checking out.

unless specifically stated otherwise, discussions herein using terms such as "processing," "computing," "calculating," "determining," "presenting," "displaying," or the like, may refer to the action or processes of a machine (e.g., a computer) that controls or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

As used herein, any reference to "some embodiments" or "teaching" means that a particular element, feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment. The appearances of the phrase "in some embodiments" or "teaching" in various places in the specification are not necessarily all referring to the same embodiment.

furthermore, the drawings depict preferred embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein

upon reading this disclosure, those skilled in the art will understand through the principles disclosed herein additional alternative structural and functional designs for the systems and methods described herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. It will be apparent to those skilled in the art that various modifications, changes, and variations can be made in the arrangement, operation, and details of the systems and methods disclosed herein without departing from the spirit and scope as defined in any appended claims.