阅读说明：本技术 用于识别用户界面元素的技术以及使用其的系统和设备 (Techniques for identifying user interface elements and systems and devices using same ) 是由 W·利德尔 P·克莱门斯 T·苏切 于 2018-05-18 设计创作，主要内容包括：本公开的实施方案整体涉及用于识别用户界面(UI)中的元素的技术,并且更具体地讲,涉及用于确定在接触敏感用户界面上选择的UI元素的技术以及使用这些技术来提供一个或多个触觉响应。(Embodiments of the present disclosure relate generally to techniques for identifying elements in a User Interface (UI), and more particularly, to techniques for determining UI elements selected on a contact-sensitive user interface and using these techniques to provide one or more haptic responses.)

1. A method of creating an instruction for searching an element of a Graphical User Interface (GUI) displayed on a touch-sensitive screen, the method comprising:

parsing a GUI definition and identifying elements of the GUI in response to the parsing;

creating a record comprising entries for the identified elements;

associating the identified element with a similarly positioned group of elements;

arranging records of the identified elements into a tree structure;

collapsing identified elements in the same group into a single leaf in the tree structure;

optimizing the tree structure; and

a list of search instructions is created in response to the tree structure.

2. The method of claim 1, further comprising creating one or more group search instructions in response to each of the groups.

3. The method of claim 2, further comprising:

grouping all identified elements by a common characteristic; and

reordering the tree structure to ensure that all identified element records can be traversed in a single pass; and

eliminating redundant or inefficient settings in response to the reordering.

4. The method of claim 3, further comprising traversing all nodes of the tree structure and, upon accessing each node, inserting one or more search instructions into a list of search instructions in response to the accessed node.

5. The method of claim 4, wherein the list of search instructions is an extensible markup language (XML) file.

6. A computer program product for enabling a computer to create executable instructions to search for elements of a Graphical User Interface (GUI), comprising:

a computer readable medium; and

software instructions on the computer-readable medium adapted to enable the computer to:

parsing a GUI definition and identifying elements of the GUI in response to the parsed GUI definition;

creating a record comprising an entry for the identified element;

associating the identified element with a similarly positioned group of elements;

arranging the records of the identified elements into a tree structure;

collapsing identified elements in the same group into a single leaf in the tree structure;

optimizing the tree structure; and

a list of search instructions is created in response to the tree structure.

7. The computer program product of claim 6, wherein the software instructions are further adapted to enable the computer to: one or more group search instructions are created in response to each of the groups.

8. The computer program product of claim 7, wherein the software instructions are further adapted to enable the computer to:

grouping all identified elements by a common characteristic; and

reordering the tree structure to ensure that all identified element records can be traversed in a single pass; and

eliminating redundant or inefficient settings from the record in response to the reordering.

9. The computer program product of claim 8, wherein the software instructions are further adapted to enable the computer to: all nodes of the tree structure are traversed and, upon accessing each node, one or more search instructions are inserted into a list of search instructions in response to the accessed node.

10. The computer program product of claim 9, wherein the list of search instructions is an extensible markup language (XML) file.

11. A microcontroller operably coupled to a touch screen configured to display a Graphical User Interface (GUI), the microcontroller comprising:

at least one processor;

one or more executable instructions stored on a non-transitory storage medium, the instructions when executed by the processor adapted to enable the processor to:

determining a location of a sensed touch at the touch screen; and

identifying a GUI element associated with a touch location corresponding to the sensed touch.

12. The microcontroller of claim 11, wherein the instructions are further adapted to enable the processor to identify a user feedback response in response to the identified GUI element.

13. The microcontroller of claim 12, wherein the instructions are further adapted to enable the processor to create one or more control signals in response to the user feedback response associated with the identified GUI element.

14. The microcontroller according to claim 13, wherein the user feedback response is a haptic response.

15. The microcontroller of claim 14, wherein the control signal is configured to control a haptic driver to excite one or more actuators in the touch screen.

16. The microcontroller of claim 15, further comprising determining a force level associated with the touch, and wherein the processor creates the one or more control signals in response to the force level.

17. The microcontroller of claim 11, wherein the one or more executable instructions comprise one or more conditional search instructions that, when executed by the processor, are adapted to enable the processor to:

comparing the location of the touch to locations of one or more elements of the GUI;

selecting a GUI element in response to the comparison; and

providing a haptic instruction and an element identifier corresponding to the selected GUI element.

18. The microcontroller of claim 17, wherein the haptic instruction is configured to instruct a haptic driver to actuate one or more actuators in a touch screen in response to the haptic instruction.

19. A method of identifying an element of a Graphical User Interface (GUI) displayed on a touch screen, the method comprising:

determining a location of a sensed touch at the touch screen;

executing one or more search instructions in response to the location, wherein each search instruction of the one or more search instructions corresponds to a GUI element and is adapted to return search results when executed; and

a GUI element is identified in response to the search result.

20. The method of claim 19, further comprising identifying a user feedback response in response to the identified GUI element.

21. The method of claim 20, further comprising creating one or more control signals in response to the user feedback response associated with the identified GUI element.

22. The method of claim 21, wherein the user feedback response is a haptic response.

23. The method of claim 22, wherein the control signal is configured to control a haptic driver to actuate one or more actuators in the touch screen.

24. The method of claim 23, further comprising determining a force level associated with the touch, and wherein the processor creates the one or more control signals in response to the force level.

25. The method of claim 19, wherein each search instruction comprises a payload inserted in response to the search tree.

26. A system, comprising:

a display subsystem configured to control a display;

a touch subsystem comprising a touch sensor and a touch controller, wherein the touch controller is configured to:

determining a location of a sensed touch at the touch sensor;

executing one or more search instructions in response to the location and search tree, wherein each search instruction of the one or more search instructions corresponds to a GUI element and is adapted to return search results when executed;

identifying a GUI element in response to the search result; and

generating a haptic control message in response to the identified GUI element.

27. The system of claim 26, further comprising a haptic feedback subsystem configured to generate haptic feedback at the display.

28. The system of claim 26, wherein some of the one or more search instructions comprise a jump instruction associated with another search instruction of the one or more search instructions, the jump instruction, when executed, configured to load the other search instruction.

29. The system of claim 28, wherein the jump instruction enables the one or more search instructions to be executed in a tree order.

30. The system of claim 26, further comprising an automotive host.

Technical Field

Embodiments of the present disclosure relate generally to techniques for identifying elements in a User Interface (UI), and more particularly, to techniques for determining UI elements selected on a contact-sensitive user interface and using these techniques to provide one or more haptic responses.

Background

Touch interfaces incorporating touch sensing are used in a variety of applications, including, for example, in tablet computers, personal computers, smart phones, and other consumer products. They are also used as control panels for automobiles, household appliances (e.g., refrigerators, ovens, washing/drying machines, etc.), heating and air-conditioning control systems, security systems, and Automated Teller Machines (ATMs). The touch interface in these applications may be, for example, a touch pad, or may include a screen and a Graphical User Interface (GUI).

In general, a touch interface with sufficient responsiveness and accuracy to be used with many applications is desired.

Disclosure of Invention

In general, some embodiments of the disclosure relate to a method of creating an instruction for searching an element of a Graphical User Interface (GUI) displayed on a touch-sensitive screen. The method includes parsing the GUI definition and identifying an element of the GUI in response to the parsing; creating a record comprising entries for the identified elements; associating the identified elements with similarly positioned groups of elements; arranging records of the identified elements into a tree structure; collapsing identified elements in the same group into a single leaf in a tree structure; optimizing a tree structure; and creating a search instruction list in response to the tree structure.

In general, some embodiments of the present disclosure relate to a computer program product for enabling a computer to create executable instructions to search for elements of a Graphical User Interface (GUI). The program product may include a computer-readable medium and software instructions on the computer-readable medium. The software instructions on the computer-readable medium are adapted to enable the computer to: parsing the GUI definition and identifying elements of the GUI in response to the parsed GUI definition; creating a record comprising entries for the identified elements; associating the identified elements with similarly positioned groups of elements; arranging records of the identified elements into a tree structure; collapsing identified elements in the same group into a single leaf in a tree structure; optimizing a tree structure; and creating a search instruction list in response to the tree structure.

Some embodiments of the present disclosure generally relate to a microcontroller operatively coupled to a touch screen configured to display a Graphical User Interface (GUI). The microcontroller includes at least one processor and one or more executable instructions stored on a non-transitory storage medium. The instructions, when executed by the processor, are adapted to enable the processor to: determining a location of a sensed touch at the touch screen; and identifying a GUI element associated with the touch location corresponding to the sensed touch.

In general, some embodiments of the present disclosure relate to a method of identifying an element of a Graphical User Interface (GUI) displayed at a touch screen. The method includes determining a location of a sensed touch at the touch screen; executing one or more search instructions in response to the location, wherein each search instruction of the one or more search instructions corresponds to a GUI element and is adapted to return search results when executed; and identifying the GUI element in response to the search result.

Some embodiments of the present disclosure generally relate to a system. The system includes a display subsystem and a touch subsystem. The display subsystem is configured to control the display. The touch subsystem includes a touch sensor and a touch controller. The touch controller is configured to determine a location of a sensed touch at the touch sensor; executing one or more search instructions in response to the location and the search tree, wherein each search instruction of the one or more search instructions corresponds to a GUI element and is adapted to return search results when executed; identifying a GUI element in response to the search result; and generating a haptic control message in response to the identified GUI element.

Drawings

Objects and advantages of embodiments of the present disclosure will be apparent to those skilled in the art from the following detailed description taken in conjunction with the accompanying drawings. The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the office upon request and payment of the necessary fee.

FIG. 1 is a swim lane diagram illustrating a process of generating and using a search task list to identify UI elements contacted in a GUI.

Fig. 2 is a flow diagram of a process for generating a search task list according to one embodiment of the present disclosure.

FIG. 3 is a flow diagram of a process for extracting UI elements from a UI structure definition according to one embodiment of the disclosure.

FIG. 4 is a flow diagram of a process of generating an intermediate search tree according to one embodiment of the present disclosure.

FIG. 5 is a flow diagram of a process of generating a search task tree according to one embodiment of the present disclosure.

Fig. 6 is a flow diagram of a process of generating a search task list according to one embodiment of the present disclosure.

Fig. 7 is a flow diagram of a process of determining whether a touch has occurred within a UI element according to one embodiment of the disclosure.

Fig. 8 illustrates an embodiment of a radio GUI composed of UI elements.

Fig. 9 illustrates UI elements of the radio GUI of fig. 8 grouped according to embodiments of the disclosure.

Fig. 10 illustrates UI elements in a tree structure formed in accordance with an embodiment of the present disclosure.

FIG. 11 illustrates an embodiment of a system containing a search task list.

Fig. 12 illustrates an embodiment of a radio GUI that includes features and parameters associated with at least some UI elements of the radio GUI.

FIG. 13 illustrates an embodiment of the system of FIG. 11 incorporated as a subsystem in the host of an automobile.

Detailed Description

In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific examples in which embodiments of the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. However, other embodiments may be utilized, and structural, material, and method changes may be made without departing from the scope of the present disclosure. The illustrations presented herein are not intended to be actual views of any particular method, system, device, or structure, but are merely idealized representations which are employed to describe the embodiments of the present disclosure. The drawings presented herein are not necessarily drawn to scale. For the convenience of the reader, like structures or components in the various drawings may retain the same or similar numbering; however, similarity in numbering does not imply that the structures or components must be identical in size, composition, configuration, or any other characteristics.

It will be readily understood that the components of the embodiments, as generally described herein, and illustrated in the figures, could be arranged and designed in a wide variety of different configurations. Thus, the following description of various embodiments is not intended to limit the scope of the present disclosure, but is merely representative of various embodiments. While various aspects of the embodiments may be presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The following description may include examples to assist those of ordinary skill in the art in practicing the disclosed embodiments of the present invention. The use of the terms "exemplary," "by way of example," and "e.g.," mean that the associated description is illustrative, and although the scope of the disclosure is intended to cover examples and legal equivalents, the use of such terms is not intended to limit the embodiments or the scope of the disclosure to the specified components, steps, features, functions, or the like.

Moreover, the particular embodiments shown and described are merely examples and should not be taken as the only way to implement the present disclosure unless otherwise indicated herein. Elements, circuits, and functions may be shown in block diagram form in order not to obscure the present disclosure in unnecessary detail. Rather, the particular embodiments shown and described are merely exemplary and should not be taken as the only way to implement the present disclosure unless otherwise indicated herein. Additionally, block definitions and logical partitioning between individual blocks are examples of particular embodiments. It will be apparent to those of ordinary skill in the art that the present disclosure may be practiced with many other partitioning solutions. In most cases, details concerning timing considerations and the like have been omitted where such details are not necessary to obtain a complete understanding of the present disclosure and are within the abilities of persons of ordinary skill in the relevant art.

Those of ordinary skill in the art would understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the specification may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. Some of the figures may show signals as a single signal for clarity of presentation and description. It will be understood by those of ordinary skill in the art that the signals may represent a signal bus, where the bus may have a variety of bit widths, and the present disclosure may be implemented on any number of data signals, including a single data signal.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a special purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor, which may also be referred to herein as a host processor or simply a host, may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. A general-purpose computer including a processor is considered a special-purpose computer when it is configured to execute computing instructions (e.g., software code) related to embodiments of the present disclosure.