阅读说明：本技术 用于改进的搜索查询相关性的无监督实体和意图标识 (Unsupervised entity and intent identification for improved search query relevance ) 是由 G·普拉萨德 V·阿帕斯瓦米 B·T·常德里 于 2020-03-17 设计创作，主要内容包括：这里描述了用于通过如下操作来尤其改进搜索查询相关性的技术：在搜索引擎上执行查询；检索从执行查询所生成的搜索页面数据,该搜索页面数据包括文档标题和通用资源定位符(URL),每个文档标题是与URL相关联的文档的标题；根据用于匹配查询、域URL和域标题中的搜索项的实体相关性得分,来确定查询中的相关实体单词；根据基于搜索项在查询和URL中相对于查询和URL中的其他搜索项的出现次数的意图单词相关性得分来确定查询中的相关意图单词；比较所确定的相关实体单词中的每个相关实体单词和所确定的相关意图单词中的每个相关意图单词与多个存储的过去用户查询；检索包括相关实体单词和相关意图单词的多个存储的过去用户搜索查询；以及向客户端设备传输查询建议集合。(Techniques are described herein for improving search query relevance, among other things, by: executing the query on a search engine; retrieving search page data generated from executing the query, the search page data including document titles and Universal Resource Locators (URLs), each document title being a title of a document associated with a URL; determining relevant entity words in the query based on the entity relevance scores for the search terms in the matching query, domain URL, and domain title; determining relevant intent words in the query from the intent word relevance scores based on the number of occurrences of the search term in the query and URL relative to other search terms in the query and URL; comparing each of the determined related entity words and each of the determined related intent words to a plurality of stored past user queries; retrieving a plurality of stored past user search queries that include related entity words and related intent words; and transmitting the set of query suggestions to the client device.)

1. A method for improving search query relevance in a system having one or more processors, one or more memories, one or more storage devices, and a query collection subsystem that receives and processes search queries sent from one or more client devices, the method comprising:

executing, by the query collection subsystem, a current user search query using an online search engine, the current user search query having a plurality of search terms;

retrieving, by the query collection subsystem, search page data generated from performing the current user search query using the online search engine, the search page data including one or more document titles and one or more Universal Resource Locators (URLs), each document title being a title of a document as follows: the document is displayed in the generated search page data and associated with one of one or more URLs;

determining, by the query collection subsystem, one or more relevant entity words in the current user search query in accordance with calculating entity word relevance scores for one or more matching search terms in the current user search query and at least one of one or more domain URLs or one or more domain titles, each domain URL being a top-level domain name and each domain title being one of the one or more document titles;

determining, by the query collection subsystem, one or more relevant intent words in the current user search query in accordance with computing an intent word relevance score based on a number of occurrences of a search term in the current user search query and the URL relative to other search terms in the current user search query and the URL;

comparing, by the query collection subsystem, each of the determined related entity words and each of the determined related intent words to a plurality of past user search queries stored in a search log;

retrieving, using the query collection subsystem, the stored plurality of past user search queries from the search log, the plurality of past user search queries including at least one of the determined one or more relevant entity words or the determined one or more relevant intent words; and

transmitting, by the query collection subsystem, a set of query suggestions for display on a user interface of a client device, the set of query suggestions being the plurality of retrieved past user search queries associated with the current user search query.

2. The method of claim 1, wherein the plurality of past user search queries are received within an adjustable tracking period.

3. The method of claim 1, wherein each of the current user search query and the past user search query is received from one or more of a human user and an automated robot.

4. The method of claim 1, wherein after retrieving the generated search page data, the method further comprises:

parsing, by the query collection subsystem, a plurality of words in the retrieved search page data; and

retrieving, by the query collection subsystem, one or more domain URLs and one or more domain titles from the search page data, at least one of the domain URLs or the domain titles including at least one exact match word for a plurality of parsed words in the retrieved search page data.

5. The method of claim 4, wherein determining the related entity words in the current user search query comprises:

determining, by the query collection subsystem, a set of entity words of the plurality of parsed words in the retrieved search page data when one or more of the parsed words are exact match words in at least one of the one or more domain URLs or the one or more domain titles;

providing a click count log of user clicks detected on each of the entity words over an adjustable tracking period, the click count log storing a number equal to click counts detected over the tracking period by a current user and a plurality of past users; and

determining, by the query collection subsystem, an entity word relevance score for each word in the set of entity words using a probability distribution that correlates to: (a) frequency of occurrence of entity words in the retrieved search page data and the current user search query, and (b) click counts detected on the entity words, the distribution applied to establish the entity word relevance score as a first probability value, the entity word relevance score identifying a related entity word when the first probability value for the entity words in the set of entity words exceeds a quantitative entity threshold level, the quantitative entity threshold level determined from the entity word relevance scores of the query suggestion set over a predetermined time period.

6. The method of claim 1, wherein the determination of the one or more relevant intent words comprises:

parsing, by the query collection subsystem, a plurality of search terms in the current user search query;

determining, by the query collection subsystem, a set of intent words in the plurality of parsed search terms when one or more of the parsed search terms are exact match words in the one or more URLs; and

determining, by the query collection subsystem, an intent word relevance score for each word in the set of intent words using a probability distribution that correlates to: (a) a number of times that one of the one or more intent words occurs in the current user search query and the URL, and (b) a number of times that each intent word occurs in the current user search query and the URL, the distribution being applied to establish the intent word relevance score as a second probability value, the intent word relevance score identifying a relevant intent word when the second probability value for the intent word in the set of intent words exceeds a quantitative intent threshold level, the quantitative intent threshold level being determined from the intent word relevance scores of the query suggestion set over a predetermined time period.

7. The method of claim 1, wherein the retrieving the stored plurality of past user queries further comprises: ranking, by the query collection subsystem, each of the retrieved past user search queries in descending order of entity word relevance score for each related entity word and in descending order of intent word relevance score for each related intent word, the ranked search queries including the set of query suggestions associated with the current user search query.

8. The method of claim 7, wherein the transmitting the plurality of past user search queries comprises: transmitting, by the query collection subsystem, the ranked search queries for display on the user interface of the client device.

9. A computer system for improved search query relevance, the system comprising:

one or more processors;

one or more secondary storage devices communicatively coupled to the one or more processors;

one or more memories from which instructions and data are retrieved by the one or more processors and in which instructions and data are stored by the one or more processors, the one or more memories having stored thereon instructions for executing a search query collection subsystem;

the search query collection subsystem is communicatively coupled to a search query database, the database storing a click count log and a search log, the search log including an index table referencing a plurality of past user search queries and related search page data for each of the plurality of past user queries, the search query collection subsystem adapted to:

executing a current user search query using an online search engine, the current user search query having a plurality of search terms;

retrieving search page data for the current user search query using the online search engine, the search page data including one or more document titles and one or more Universal Resource Locators (URLs), each document title being a title of a document associated with one of the one or more URLs;

determining one or more relevant entity words in the current user search query in accordance with calculating entity word relevance scores for the plurality of domain URLs, the domain title, and one or more matching search terms in the current user search query;

determining one or more relevant intent words in the current user search query in accordance with computing an intent word relevance score based on a number of occurrences of a search term in the current user search query and the URL relative to other search terms in the current user search query and the URL;

comparing each of the determined related entity words and each of the determined related intent words to the plurality of past user search queries referenced in the search log;

retrieving the referenced plurality of past user search queries, the referenced plurality of past user search queries including at least one of the determined one or more related entity words or the determined one or more related intent words; and

transmitting a set of query suggestions for display on a user interface of a client device, the set of query suggestions being the plurality of retrieved past user search queries associated with the current user search query.

10. The system of claim 9, wherein the plurality of past user search queries are stored in the search log over an adjustable tracking period.

11. The system of claim 9, wherein the current user search query and each of the past user search queries stored in the search log are received from one or more of a human user and an automated robot.

12. The system of claim 9, wherein the search query collection subsystem is further adapted to:

parsing the retrieved plurality of words in the search page data; and

retrieving one or more domain URLs and one or more domain titles from the search page data, at least one of the domain URLs or the domain titles comprising at least one exact match word for one or more of the parsed words in the retrieved search page data, each domain URL being a top level domain name, and each domain title being one of the one or more document titles.

13. The system of claim 12, wherein the search query collection subsystem is further adapted to:

determining a set of entity words of the plurality of parsed words in the retrieved search page data when one or more of the parsed words are exact match words in at least one of the one or more domain URLs or the one or more domain titles;

detecting user clicks on each of the entity words over an adjustable tracking period, the click count log storing a number equal to click counts detected over the tracking period by a current user and a plurality of past users; and

determining an entity word relevance score for each word in the set of entity words using a probability distribution that relates to: (a) a frequency of occurrence of entity words in the retrieved search page data and the current user search query, and (b) the click count detected on the entity words, the distribution applied to establish the entity word relevance score as a first probability value, the entity word relevance score identifying a related entity word when the first probability value for the entity words in the set of entity words exceeds a quantitative entity threshold level, the quantitative entity threshold level determined from the entity word relevance scores of the query suggestion set over a predetermined time period.

14. The system of claim 10, wherein the search query collection subsystem is further adapted to:

parsing a plurality of search terms in the current user search query;

determining a set of intent words in the plurality of parsed search terms when one or more of the parsed search terms are exact matching words in the one or more URLs; and

determining an intent word relevance score for each word in the set of intent words using a probability distribution that is related to: (a) a number of times that one of the one or more intent words occurs in the current user search query and the URL, and (b) a number of times that each intent word occurs in the current user search query and the URL, the distribution being applied to establish the intent word relevance score as a second probability value, the intent word relevance score identifying a relevant intent word when the second probability value for the intent word in the set of intent words exceeds a quantitative intent threshold level, the quantitative intent threshold level being determined from the intent word relevance scores of the query suggestion set over a predetermined time period.

15. The system of claim 9, wherein the search query collection subsystem is further adapted to:

ranking each of the retrieved past user search queries in descending order of entity word relevance score for each related entity word and in descending order of intent word relevance score for each related intent word, the ranked search queries comprising a set of query suggestions associated with the current user search query; and

transmitting the ranked search query for display on the user interface of the client device.

Technical Field

The present disclosure relates generally to the field of computing and communications, and particularly, but not exclusively, to unsupervised entity and intent identification in search queries for improved search query relevance on online search engines accessed over computer communication networks.

Background

The rapid development of search engine technology has created opportunities for users seeking information from online accessible sources to provide enhanced services. The primary means of finding information is to utilize online search engines such as those available on bing. Additionally, embedded search engines with online services and social media platforms (such as) The advent of the technology has accelerated the need for high speed, accurate and precise search capabilities. In fact, the large amount of online available information has spawned an increasing demand for online assistants that can help users refine their searches quickly, while preserving the relevance of search queries in an effort to identify online resources, such as documents, videos, images, real-time audio content, and recorded audio content, etc., that are more relevant to their search queries than would be possible without such rapid refinement.

The dynamic generation of such search query refinements ultimately requires some level of understanding of the user's intent. In many cases, the user's actual intent may be represented in the multi-word query in one or a few words (e.g., celebrity name, brand, service, product or action, etc.). However, few online search systems are able to efficiently determine the user's intent without some a priori knowledge. Also, the acquisition of a priori knowledge often requires that certain words or terms be pre-labeled, that the semantic scope of the query be limited to a particular topic, or that the user be forced to provide more detailed information in the search query to make a more structured description of the search intent more apparent. Indeed, among currently used options for refining search queries, including auto-suggestions and related search capabilities, there are at least three major problems in attempting to understand user intent from a search query, as they are often presented online. These problems include: the lack of available tagged data to help infer the user's intent; structured data is lacking when presented in queries because they are often typed in by users, taking little or no consideration of formal grammar rules; and sparsity of data in the query, which makes it difficult or impossible for an online system to learn the usage patterns of words in the query in a completely unsupervised manner.

Accordingly, there is a significant and growing need for robust systems and methods for improving search query relevance using unsupervised methods to quickly identify intent or entity words in a search query and use those words to identify and retrieve previously relevant search queries and display them as candidate query suggestions to a current user of a search engine in order of relevance rank.

Disclosure of Invention

A method and system for unsupervised entity and intent identification in search queries is described for improved search query relevance using an online query collection subsystem to capture and evaluate user search queries submitted to search) engines, such as bing. One key goal of the system and method is to identify and extract such entities and intent words as a means of identifying previously received user search queries (i.e., past user queries) that include the same or semantically similar entities or intent words. Once identified in the set of past user queries, the queries are retrieved from search logs accessed by the query collection subsystem and displayed as alternative query suggestions in a relevance ranked order on a user interface of a device for submitting the search queries to a search engine.

In determining which past user search queries are related to a received search query executing on a search engine, words in a search engine results page ("SERP") retrieved from executing the search query on the search engine are parsed while a process of identifying, extracting, and retrieving web addresses (i.e., universal resource locators or "URLs") and document titles in the SERP is performed. The words in the SERP that exist in the top level domain name in the web address or document title are initially extracted and compiled in an "entity" word list. These entity words are further processed to determine whether each entity word is also present in the received search query. Additionally, a historical number of clicks and frequency of occurrence received on each of the physical words identified in the collection of search engine result pages within the scrolling time window of the collected queries is determined. In general, the presence of a solid word, its frequency of occurrence, and the historical click count on the solid word are used to calculate a probabilistic relevance score and compare it to a quantitative "solid" word threshold level. When the search logs are used to identify and retrieve past user queries having the same or similar relevance scores, entity words having probability scores greater than the threshold level are considered "relevant" and are subsequently used as higher priority terms.

The systems and methods disclosed herein are used to perform a similar process to determine which words in the current search query are intent words, and which of these words are "related" intent words. The list of URLs is retrieved from a SERP generated by performing a search using a received search query from a search engine. In addition, each word of the search query is parsed by the query collection subsystem and compared to the URLs in the SERP results. The SERP words located in the URL are added to the list of identified intent words, and then the list is further evaluated to determine which of the identified intent words are "related" intent words to be assigned a probability score that indicates intent relevance. As with the entity words, an intention word is considered relevant if and only if its calculated probabilistic relevance score exceeds a quantitative intention threshold level, which is different from the quantitative entity level previously described.

Once calculated, the individual word probability scores are used as relative weights to identify past user searches stored in the search logs having the same or similar quantitative probability profiles. The quantitative scores are then used to rank the various past user searches in the relative order of relevance, which are then retrieved, ranked in order, and displayed to the search engine user as suggested queries. Such suggested queries may be generated in a completely unsupervised manner, without the need for pre-tagging data or words, and may be generated from completely unstructured user input (i.e., search queries) ranging from phrases to grammatically incorrect phrases or sentences.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Drawings

Non-limiting and non-exhaustive embodiments are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

FIG. 1 is a block diagram illustrating an operating environment of a system that provides improved search query relevance in an embodiment.

FIG. 2 is a block diagram illustrating an embodiment of a client device for use with a system for providing improved search query relevance.

FIG. 3A is an illustration of a query database in a system for improved search query relevance in an embodiment.

FIG. 3B is a flow diagram illustrating an embodiment of a method for storing search queries and search result data for improved search query relevance.

FIG. 4 is an illustration of a system for improved search query relevance in an embodiment.

FIG. 5 is a flow diagram illustrating an embodiment of a method for improved search query relevance in an embodiment.

FIG. 6A is a flow diagram illustrating an embodiment of a method for identifying entity words for improved search query relevance.

FIG. 6B is a flow diagram illustrating an embodiment of a method for determining related entity words for improved search query relevance.

FIG. 7A is a flow diagram illustrating an embodiment of a method for identifying intent words for improved search query relevance.

FIG. 7B is a flow diagram illustrating an embodiment of a method for determining relevant intent words for improved search query relevance.

FIG. 8 is a flow diagram illustrating an embodiment of a method for generating a set of query suggestions for improved search query relevance.

FIG. 9 is a flow diagram illustrating an embodiment of a method for ranking a set of query suggestions for improved search query relevance.

Detailed Description

The following example embodiments relate to a system and method for unsupervised entity and intent identification for improved search query relevance. The examples described below allow a user to enter a free-form search query on a search engine, generate a search results page that is relevant to the subject matter of the search query, and receive a ranked list of relevance order of query suggestions based on a particular entity or intent in the user search query that is the same or similar to the entity or intent words in past user search queries. Relevance is determined based on the degree of similarity of entities or intent words between the user's current search query and past user search queries stored in the aggregated search log. The past search queries with the highest level of relevant similarity are ranked higher in an ordered ranked list of query suggestions retrieved from search logs that are dynamically updated within an adjustable tracking period.

A technical effect of the embodiments described below relates to determining and retrieving, in an unsupervised manner, past search queries that are relevant to a user's actual intent in a current search query provided to a search engine. Collectively, these embodiments cause past user search queries that are more relevant to the subject matter of the current user search query to be identified and retrieved in a timely manner when compared to current alternatives such as auto-suggest features or other contemporary relevant search capabilities.

As described more fully below, an example computing system determines a set of query suggestions relevant to a user's search query based on identification and evaluation of significant words in a search engine results page and the received user search query. These so-called important words are either entity words or intention words. Upon determining that the words are entities or intent words, the system parses a search engine result page that results from executing the received search query on a search engine, and initially proceeds to identify parsed words that appear in the document titles and in the top level domain names of the search engine result pages that are relevant to the received search query. Resolved words that appear in the top level domain name, document title, or both are classified as either entity or intent words. Thereafter, the system further proceeds to determine the frequency of occurrence of the entity words or the intent words, and then determine whether any of these words occur in the received search query. Based on two separately calculated probabilistic relevance scores, one being an entity word relevance score and the other being an intention word relevance score, parsed words that appear in the top level domain name or document title at a significant level of historical click count and also appear in the received search query with sufficiently high frequency are categorized as "related" entity words or intention words. Once categorized, the system continues to identify past user search queries stored in or accessible through the aggregated search log that have the same or similar relevance scores as the entities or intent words to which the received search query is related. Past user queries having relevance scores that are the same as or similar to the probabilistic relevance score calculated for the received search query are retrieved and displayed on the user interface of the client device as query suggestions related to the received search query in a relevance ranking order.

As a preliminary matter, some of the figures describe concepts in the context of one or more structural components, variously referred to as functions, modules, features, elements, and so forth. The various components shown in the figures may be implemented in any manner, such as software, hardware, firmware, or a combination thereof. In some cases, the various components shown in the figures may reflect the use of corresponding components in an actual implementation. In other cases, any single component illustrated in the figures may be implemented by multiple actual components. The depiction of any two or more separate components in the drawing figures may reflect different functions performed by a single actual component.

Other figures describe these concepts in flow chart form. In this form, certain operations are described as constituting distinct blocks performed in a particular order. Such implementations are examples and non-limiting. Some of the blocks described herein may be combined and performed in a single operation, some blocks may be broken down into multiple component blocks, and some blocks may be performed in an order different than illustrated herein, including the manner in which the blocks are performed in parallel. The blocks shown in the flow diagrams may be implemented by software, hardware, firmware, manual processing, and the like. As used herein, hardware may include microprocessors, Digital Signal Processors (DSPs), microcontrollers, computer systems, discrete logic components, and/or custom logic components, such as Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Programmable Logic Arrays (PLAs), and the like.

With respect to terminology, the phrase "configured to" encompasses any manner in which any type of functionality may be constructed to perform an identified operation. The functionality may be configured to perform operations using, for example, software, hardware, firmware, etc. For example, the phrase "configured to" may refer to a logical circuit structure of hardware elements arranged to perform an associated function. The phrase "configured to" may also refer to a logical circuit structure of hardware elements arranged as a coded design implementing the associated functionality of firmware or software. The term "module" refers to a structural element that may be implemented using any suitable hardware (e.g., processor, etc.), software (e.g., application, etc.), firmware and/or any combination of hardware, software, and firmware. The term "logic" encompasses any functionality for performing a task. For example, each operation illustrated in the flowcharts corresponds to logic for performing the operation. Operations may be performed using software, hardware, firmware, or the like. The terms "component," "system," and the like can refer to a computer-related entity, hardware, and software in execution, firmware, or a combination thereof. A component may be a process running on a processor, an object, an executable, a program, a function, a subroutine, a computer, or a combination of software and hardware. The term "processor" may refer to a hardware component, such as a processing unit of a computer system.

Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computing device to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to encompass a computer program accessible from any non-transitory computer-readable storage device or media. The computer-readable storage medium may include, but is not limited to, magnetic storage devices such as hard disks, floppy disks, magnetic strips, optical disks, Compact Disks (CDs), Digital Versatile Disks (DVDs), smart cards, flash memory devices, and the like. In contrast, computer-readable media, i.e., non-storage media, may additionally include communication media such as transmission media for wireless signals and the like.

FIG. 1 is a block diagram illustrating an exemplary embodiment 100 of a system and method of unsupervised entity and intent identification for improved search query relevance. The environment 100 includes an online system 103 and client devices 105a and 105b connected via a network 101. Although a selected number of each device is shown in fig. 1, embodiments may have more or fewer of each device (e.g., additional client devices 105, etc.).

The online system 103 is comprised of interoperating computer hardware resources and computer software subsystems that provide query collection and query relevance ranking services to users. In one embodiment, the online system 103 includes a query database, an entity subsystem, an intent subsystem, and an interoperability module for identification and ranking of related past user search queries related to a newly received user search query. The online system 103 collects each new user query, retrieves the search engine results page relevant to the query, and performs parsing of the query and words in the search engine results page to identify intent words or entity words. The database provided in online system 103 includes not only a running log of past user queries, but also a log of click count activity on words of a search engine results page associated with each past user query. The online system 103 combines the data provided from the parsed words, the historical click counts on the parsed words, and the absolute and relative frequency of occurrence of the parsed words to identify entity words or intention words, which are then scored, relevance ranked, and used to retrieve past user queries having the same or similar relevance scores as those computed from the word parsing of the query and search engine results pages. Past user queries having relevance scores greater than a predetermined threshold level for an entity or intent word are then transmitted by the online system 103 for display on the user interface of the client device 105.

The client devices 105a, 105b are devices used by users to communicate with the online system 103. The client devices 105a, 105b may be, for example, desktop computers, laptop computers, smart phones, tablet computers, or Personal Digital Assistants (PDAs). The user communicates with the online system 103 through the client devices 105a, 105b to use the query collection subsystem in the online system 103. In response to executing a search query provided from a client device 105a, 105b, the query collection subsystem stores the query, parses words of a search engine results page generated using the search engine, determines related entity words and related intent words related to the query and the search engine results page, and retrieves past user search queries related to one or more from a database accessed by the query collection subsystem of the online system 103. The related past user queries are ranked in order by the query collection subsystem, then transmitted in a related ranked order and displayed on the user interface of the client device 105a, 105 b.

Network 101 represents a computer communication path between online system 103 and client devices 105a, 105 b. In one embodiment, network 101 is the internet and uses standard communication technologies and/or protocols. Network 101 may include links using technologies such as ethernet 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, Long Term Evolution (LTE), Digital Subscriber Line (DSL), Asynchronous Transfer Mode (ATM), InfiniBand, PCI express advanced switching, and so on. Similarly, networking protocols used on network 101 may include multiprotocol label switching (MPLS), transmission control protocol/internet protocol (TCP/IP), user datagram protocol (UCP), hypertext transfer protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), and other competing alternatives.

Data exchanged over network 101 may be represented using techniques and formats such as hypertext markup language (HTML), extensible markup language (XML), and the like. Further, all or some of the links may be encrypted using conventional encryption techniques such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), internet protocol security (IPsec), and the like. In alternative embodiments, data communication may occur using custom and/or dedicated data communication techniques, either in lieu of or in addition to those described above.

FIG. 2 illustrates a block diagram of the operational components in an embodiment of a client device 200 for use with the online system 103. In practice and as previously described, the client device 200 may be a desktop computer, laptop computer, smartphone, tablet computer, Personal Digital Assistant (PDA), or other device suitable for a human user to enter search queries and review query suggestions. However, it is important to note that the input of search queries from the client device 200 is not limited to human users. Search queries may be entered using automated robotics or other remote or distributed computational intelligence, most of which comes from machine learning and deep learning methods and systems. Thus, it is contemplated that submission of search queries and interpretation of query suggestions can be performed using such automated robotics and/or distributed intelligence systems for the purpose of identifying and performing most relevant searches based on extracted entity words and intent words.

In the illustrated embodiment, the client device 200 is coupled to a network 101, receiving data from the online system 103 and transmitting data to the online system 103 through the network 101. Indeed, multiple client devices 200 may be used to send and receive data to and from the online system 103 for identifying and retrieving relevant query suggestions from past user search queries. Data from the network 101 is received over the network interface 202 and placed in a queue for retrieval by the central processing unit 206. In this embodiment, the central processing unit 206 may be a general purpose computer or an Application Specific Integrated Circuit (ASIC), or a computing architecture adapted to use and/or process strings, lists, or other data structures suitable for maintaining relative relationships between data items. The data presented in one embodiment is a ranked ordered list of query suggestions received from the online system 103 in response to a user submitting a search query from a client device 200. In transmitting a search query to the online system 103, user input, such as a search query, is provided using an input device 214 (e.g., a monitor, touch screen display, etc.) and then received on the input/output interface 204 for queuing and transmission to the online system 103 through the network interface 202 using the central processing unit 206. In one embodiment, query suggestions transmitted from the online system 103 are received on the network interface 202 as lists or other data structures that preserve relative relationships between data items (e.g., rings, stacks, B + trees, etc.) and stored in program memory 208 (e.g., RAM, DRAM, SRAM, NVRAM, SDRAM, DDRX RAM, etc.) or on mass storage devices 210 (e.g., hard drives, floppy disks, CD-ROM, tapes, disks, drums, etc.) for retrieval by the central processing unit 206 and display on an output device 212 (e.g., monitor, flat panel display, refreshable braille display, etc.) as an ordered list for end-user review and selection.

FIG. 3A is a diagram of a query database of a query collection subsystem in online system 103, in an embodiment. In the illustrated embodiment, the query database 300 is comprised of a single database, or in an alternative embodiment, a plurality of distributed databases. Query database 300 stores aggregated search log 302 and click count log 306. Aggregated search log 302 may be implemented as a lookup table or index table, such as a hash table, or other suitable data structure for storing user search queries 304 and related search engine results pages 308 (interchangeably referred to as SERPs). In one embodiment, the stored search engine results page 308 is search results resulting from executing a search query within an adjustable tracking period, which in the illustrated embodiment is a tracking window trailing 12 months. In addition to search results, the user search query 304 is also retained and stored for an adjustable tracking period. Query 304 includes search queries from users searching for information about various topics on search engines (e.g., bing. com, yahoo. com, etc.), and they are stored for tracking user activity and for providing a repository of prior search activity, which may be used to identify appropriate query suggestions for subsequently received user search queries that may be relevant to the topic of such subsequently received search queries.

The repository of stored search queries 304 includes single word and multiple word search queries, each of which, when executed by a search engine, generates a search engine results page 308. The search results may include documents, images, videos, or other content that are directly or indirectly related to the subject matter of each search query in the stored query list 304. In addition to the aggregated search log 302, the query database 300 also includes a click count log 306 that maintains a running log of user clicks on a cumulative set of search engine results pages 308 stored in the aggregated search log 302 over an adjustable tracking period. In one embodiment, the click count log 306 is a running historical click count log that reflects user activity, particularly user click activity, related to the query 304 stored in the aggregated search log 302 and on search results that appear in a search engine results page 308. The click count is implemented in one embodiment as an extension of the aggregated search log 302 in separately identifiable fields and records maintained within the same index table, or, in an alternative embodiment, as a separately maintained but simultaneously updated association table that preserves the logical relationship between click count activity and associated search queries.

FIG. 3B is a flow diagram illustrating an embodiment of a process 300 for capturing, storing, and maintaining search queries and related search engine results pages in a query database 300. Upon receiving the user search query, as shown at step 310, the SERP generated at the search engine from executing the search query is stored in an aggregated search log, as shown at step 312. Click tracking tools embedded on web pages used by search engines as SERPs in response to execution of search queries enable monitoring of click activity on different search engine result pages, as shown at step 314, and results produced using the tools enable compiling and storing of click counts on search engine result pages associated with any search query. In particular, the process performs successive calculations and determinations of click counts on uniform resource locators ("URLs") and document titles retrieved as search results with SERPs, as shown at step 316. The search query, each of its associated universal resource locators, and the click count on each URL and each document title in the SERP are stored in turn in the aggregated search log 302 in the query database 300, as shown at step 318. In one embodiment, the aggregated search log 302 is an index table that stores each query, associated SERP, and click-through counts on URLs and document titles in the associated SERP during an adjustable tracking period. In particular, in an embodiment, the index table is a hash table that includes a hash flag that marks or identifies each query in the hash table. Separately, the query collection subsystem in online system 103 performs continuous event-driven or real-time updates and refreshes of the data and references in the index table so that saved queries, SERPs, and click counts for URLs and document titles are recorded in the search results stored within an adjustable tracking period.

FIG. 4 illustrates an embodiment of the operational components of the query collection subsystem 400 in the online system 103. As illustrated, subsystem 400 includes a query database 300 in which an aggregated search log 302 and a click count log 306 (not shown) are stored. In an embodiment, the data stored in the query database 300 is accessed by the entity subsystem 402 and the intent subsystem 420. The entity subsystem 402 in this embodiment is comprised of two interoperating modules, an entity identification module 412 and an entity relevance scoring module 414. In the depicted embodiment, the entity identification module 412 identifies "entity" words and performs word parsing on the search engine results page associated with each received search query. The identification module 412 also performs a correlation process on the SERP associated with the search query to identify a top-level domain name, referred to as a "domain URL," and each document title, referred to as a "domain title," associated with the identified domain URL. In an embodiment, the entity relevance scoring module 414 determines the relevance of each word identified using the entity identification module 412. And, in particular, the entity relevance scoring module determines whether the identified words are "entity" or "intent" words, confirms whether such words appear in the received user search query through word parsing, and if so, determines the frequency of occurrence of such words in the set of search engine result pages stored in the aggregated search log and the number of clicks received on the URLs and document titles that include such words in the collection set including the user query and associated search engine result pages. In this manner, entity words or intent words that have a high frequency of occurrence and a high number of clicks on related URLs and document titles in a search engine results page determine whether such entity or intent words are "relevant". The word is quantitatively determined to be "relevant" based on the computation of a posterior (posteroir) probability distribution function that is related to: the presence or absence of a word in the search query and in the search engine results page in the accumulated set of stored search results that extends over an adjustable tracking period. Once calculated, the probability values resulting from applying the probability distribution function to the applicable data set are compared to quantitative thresholds, one threshold being determined for "solid words" and a different threshold being determined for "intended words". Calculated probability values greater than a threshold level of entity are classified as "related" entity words, while intent words having probability distribution values greater than a threshold level of intent are classified as "related" intent words. A word in a search query is considered an "entity" word if the word represents the object or context of the action represented by the intent word. The "intent" word represents an action that a user attempts to perform on an object as an entity, or an action performed on or for the object. The quantitative entity threshold level is updated at least once every half-year period based on the calculated probabilistic relevance scores for the entity words in the SERP and the continuous manual review of the user query received over the time period. During these six-month periods, each month a sample set of search queries, related query suggestions, and computed entity word relevance scores are compiled into query suggestion triples (queries, suggestions, relevance scores). For each triple, the query-suggestion pair ("qs pair") is manually reviewed and judged as "good" or "bad" depending on the relevance score. Based on such a determination, a new threshold is selected over a predetermined period of time (in this example, every six months) to eliminate a number of lower scoring qs pairs. Thus, as the threshold level increases, the number of lower scoring bad qs pairs that affect the threshold decreases.

Often, the search query provided on a search engine is a request for information on a particular topic. The topic may also include a number of related sub-topics that may be of interest to the user providing the search query. When performing a search using a particular search query, a user may seek to reformulate or refine the query to obtain search results that are relevant to one or more sub-topics. Thus, in helping users accomplish the task of reformulating or refining search queries, it is important to know the user's implicit intent as best as possible as determined from the words provided in the initial search query. Semantically similar or related search queries from contemporaneous or previously provided search queries can be readily identified and presented as alternative search query suggestions if the user intent can be effectively determined from the words used in the initial search query.

Conceptually, each topic in a search query includes words that may be assigned different relevance weights. Related topics may have common words while unrelated topics may have infrequently shared words. For example, search queries about a topic such as "sports" are more likely to have more common words than search queries about a completely unrelated topic such as "politics". Also, for each topic, each word may have a different relevance weighting or vary widely based on their respective frequency of occurrence of relevance to the topic of the search query. Based on this basic assumption, for a given topic in a search query, the topic itself will affect the number and relevance of words used in the search query. In this manner, the core words in the search query that are more relevant to a topic, and therefore more strongly associated with a topic and more representative of the search query, may be more important, and this importance is reflected in the relative number of times a word appears in the query and the number of times the user clicks on the query, including words that are more relevant to the core of the search query.

The relationship between the search query topic and the words used in the search query and their SERPs can be represented as a word distribution that can be modeled using a multi-term distribution. In an embodiment, a composite probability distribution, referred to as a Dirichlet (Dirichlet) polynomial distribution, is used to represent or model the distribution of words present in a search query and its SERP based on the topic of the query. This form of probability distribution assigns a weight to each word based on the association of each word in the query and its SERP with the topic (i.e., the frequency of occurrence of the word). In one embodiment, the query is a topic and the words in the SERP are used to measure the frequency of occurrence of certain important words, called entity words and intent words. Two different but related models are used to determine the relevance weights for the entity words and the intent words, as illustrated below.

Assume that for each search query Q, over the last twelve (12) months on the SERP, K different algorithm results have historically been shown. Each of the K algorithm results contains a URL, a document title, and a document description. Further, assume that the URLs are domain URLs (as described above) and that a document title is associated with each such domain URL and is referred to as a domain title (as described above). It is assumed that there is a probability distribution of words associated with each search query Q. Let P^QIs in the word w_iAnd wherein is the probability distribution associated with the search query Q, and wherein is each word w_iNumber of occurrences in the list of unique URLs and associated document titles. If P is assumed^QCan be described as following a dirichlet probability distribution, then:

whereinα＝(α₁，…，α_K) And p is_wiIs the word w_iThe probability of (c). In this context, the probability P^QIs a prior probability distribution of words corresponding to the search query Q.

In addition to determining the form of the probability distribution to be applied to words in search query Q, historical click data must also be considered to further refine each word w given search query Q_iLikelihood or probability of. Such probability and probability distribution P^QUsed in combination to determine the word w_iIs adjusted to the probability p_w. In such an example, p_wIs directed to the word w_iProbabilistic relevance for entity wordsAnd (6) scoring. More specifically, w, which is used to calculate as a solid word based on historical click data_iThe expression of (1) is:

where N is the total number of words, w_iIs the (i) th word or words,is to contain the word w_iThe jth domain URL of (1) and the click count of the domain title. If it is assumed that m different URLs contain w_iThen is equal to w_iThe corresponding total number of clicks isUsing a priori probabilitiesThe likelihood of a word may be expressed as:

wherein C is a constant, and C is a constant, the full expression becomes:

and, in the respect of p_w' s optimizes L: (p|cα), the probability relevance score of an entity word can be expressed as:

the probability that an identified entity word is a "related" entity word requires that for a given w_iFor i ═ 1 to N, based on the threshold T_eIf p is_w＞T_eThen w_iAre considered not only to be physical words, but are also more formally referred to as "related" physical words. If more than one word satisfies this attribute, the relative weight or "relevance" of each word is represented by a probabilistic relevance score p_wTo decide. Threshold value T_eAre generic and determined empirically, but are manually adjusted on a semi-annual basis based on the identified entity words collected in the search engine results page and search query over this period of time.

In embodiments, the intent subsystem 420 identifies "intent" words in the search query. In this embodiment, the intent identification module 422 identifies a word in the search query as an intent word if the word in the search query is present in the user query and in a set of URLs associated with the search query. The identification module 422 identifies the intent word as a "relevant" intent word if the ratio of (i) the number of times a word appears in the associated URL and search query, compared to (ii) the number of times each word in the user query appears in the user query and associated list of URLs, is greater than a predetermined intent threshold level. The intent threshold level is determined from periodic manual review of the probabilistic relevance scores computed on the received search queries and SERPs that occur no less than once every six months. During this time period, each month a sample set of search queries, related query suggestions, and calculated intent word relevance scores are compiled into query suggestion triples (queries, suggestions, relevance scores). For each triple, the query-suggestion pair ("qs pair") is manually reviewed and judged as "good" or "bad" depending on the relevance score. Based on such a determination, a new threshold is selected over a selected period of time (in this example, every six months) to eliminate the maximum number of lower scoring qs pairs. Thus, as the threshold level increases, the number of lower scoring bad qs pairs that affect the threshold decreases. Quantitatively, the relationship between the various variables used to determine the probability that a word is an intended word is as follows:

wherein n is_iIs the word w_iNumber of occurrences in all URLs and received search queries, q_wIs w_iIs the probability of an important (or "relevant") intended word. After optimization, this relationship becomes:

in one embodiment, the intent relevance scoring module 424 establishes a quantitative relevance of the intent words in the received search query by comparing the identified intent words to an empirical threshold level. If these values are greater than the intention threshold, then for a given intention threshold T_IAnd p_wQ may be_wAre considered relevant "intent" words in the search query. As previously described with respect to T_eDescribed, intention threshold T_IIs empirically determined and is manually adjusted at least once every half year based on the method's execution on the search engine results page and search query received during the time period. In general, the "relevance" of words in a search query Q is expressed as a pair (p) for each word in the search query_w,q_w) Is determined by the combined probability correlation score of (a).

In one embodiment, the query set identification module 430 identifies queries that include one or more of related entity words and related intent words from a stored set of past user queries received over a trailing 12-month tracking window. In particular, the module 430 identifies past queries having words with a probabilistic relevance value greater than an applicable quantitative entity threshold level or quantitative intent threshold level, such that a subset of queries stored in the query database may be retrieved and listed item-by-item as potential query suggestions for the received user query. In one embodiment, the query ranking module 440 establishes a relevance ranking order for each identified query identified by the query set identification module 430 and listed on an item-by-item basis. The ranking module 440 determines a collective relevance rank for the stored queries by first establishing a relevance rank for the identified entity word (or intent word) having the highest probabilistic relevance score. In one embodiment, each query that includes this highest relevance score may be retrieved and then further classified based on the relative relevance score of each successive word in each query.

As an illustrative example of a probabilistic relevance score ranking ordering of query suggestions, assume that a search query Q is received in the form of "A B C₁Where each letter is a word. If the probability correlation score p_wAnd q is_wWhere the entity word relevance score is 0.7 and B0.5, and the intention word relevance score for C is 0.2, then Q is retrieved from the aggregated search log 302 using the set identification module 430₁The query suggestion query of (a) may be: "A D E", "A C E", "A B C D", "B C X", "B D", "C N P", where each letter is a word. The query ranking module 440 will then proceed to reorder the selected queries in the query set according to the relevance rank order, which in the illustrated example would be: "A B C D", "A C E", "A D E", "B C X", "B D", "C N P". Relevance rank ordering is determined primarily, but not exclusively, from the calculated relevance scores (intent and entity), and if a word has a higher calculated intent word relevance score (i.e., greater than the illustrated entity word relevance score), it results in a different rank ordering based on the relevance scores calculated for the word in the search query. Once the rank ordering is determined, a relevance-ordered listing is transmitted to the client device, where the relevance-ordered listing of query suggestionsDisplayed on a user interface of the client device or otherwise made available to an automated robot or other automated service or capability executing on the client device.

FIG. 5 is a flow diagram illustrating an embodiment of the operational flow of the query collection subsystem in the online system 103. Process 500 begins with the receipt of a user query, as shown at step 502, followed by step 504 of retrieving from an aggregated search log, and one embodiment concurrently executes a process for identifying certain words in the received user query as "entity words," as shown at step 506, or identifying certain words identified in the received search query as "intent words," as shown at step 510. Once one or more entity words are identified from the parsing of the words in the search query and associated search engine results page, a separate process is initiated to determine the relevance of each entity word, which involves the determination of an entity word relevance score, which is a value on the computed probability distribution, as shown at step 508. Likewise, a separate process is simultaneously initiated and performed to determine the intent word relevance score for each identified intent word, as shown at step 512. Each intended word (if present in the search query) is determined from parsing and extraction of the word in the received search query and its associated URL. The parsed and extracted words determined to be "intent words" are further processed to determine the frequency of occurrence of such words in the SERP and search queries and the click count history for each word. An intention word or an entity word having an overall probabilistic relevance score greater than some preset threshold level for the intention word or entity word is considered a "related" intention word or a "related" entity word. Once the related entity words and the related intent words are determined, a combined entity/intent relevance score is determined for each word in the search query, as shown at step 514, and then used to retrieve and rank past queries identified as related query suggestions stored in the aggregated search log, as shown at step 516. The ranked list of past user queries according to their calculated relevance to the relevant entity words and the relevant intent words in the received search query is then transmitted to the client device for display as an ordered ranked list of query suggestions. As discussed previously, the list of related query suggestions is based not only on the determination of related entity words or related intent words, but also on the frequency of occurrence of these words in the set of search engine result pages stored in the aggregated search log, the presence of words in the received search query, and the historical number of clicks (i.e., click activity) on and associated with the received search query on the search engine result pages stored in the aggregated search log, as well as past user search queries, where such past search queries and result pages include related entity words or related intent words, such that the probability relevance score for each query on the list of suggestions exceeds a quantitative entity threshold level or a quantitative intent threshold level.

FIG. 6A is a flow diagram illustrating an embodiment of a process for identifying entity words. The process 600 begins with the receipt of a user search query, as shown at step 602, followed by the retrieval of a search engine results page associated with the search query, as shown at step 604. Both the search queries and the associated search engine results pages are stored in the aggregated search log and accessible by the query collection subsystem 400 for word parsing and determination of associated frequency of occurrence and click count evaluations. The process 600 continues with retrieving the associated domain URL and domain title in the retrieved search engine results page, as shown at step 602. Concurrent with retrieving such domain URLs and domain titles is the parsing of the search engine results page, as shown at step 608. Parsing of words in the search engine results page is performed to identify and evaluate common words (including those resulting from execution of the received search query on a search engine) between the received search query and the complete set of search engine results pages. Once the word parsing is performed, each parsed word is compared to the identified domain URL and domain title, as shown at step 610. If the parsed word is in the associated domain URL or domain title, then the word is added to the itemized list of identified entity words, as shown at step 614. Conversely, if the word is not in the associated domain URL or domain title, then it is considered not a solid word, as shown at step 612.

In FIG. 6B, an embodiment of a process 600 is depicted that begins with the generation of a click count log based on click activity on URLs associated with each search query, as shown at step 616. When click activity is determined, the click count log stored in the query database is continually updated so that a continuously running log of click count history is stored for an adjustable tracking period. In addition to determining the historical click count of words in the saved search engine results page and in the search query, the frequency of occurrence of each entity word in the itemized list is also determined, as previously described and as shown at step 618. The frequency of occurrence of entity words is determined over a pool of words included in a set of received search queries, their associated search engine results pages, and search engine results pages and queries stored in an aggregated search log. After generating the list of identified entity words and determining the frequency of occurrence and the number of clicks for each entity word, a probability distribution relevance score is calculated to determine whether the identified entity word is a "related" entity word, as shown at step 620. The calculation of the entity word relevance score requires the application of a probability distribution (such as a dirichlet distribution in one embodiment) to the set of words in the search query from which the entity words are identified and extracted. The entity word relevance score is a probability value over the applied probability distribution. After calculating the relevance scores, additional searches are performed to confirm that the identified entity words are also terms in the received user search query, as shown at decision step 622, as a first step in determining that the identified entity words are "relevant" entity words. After determining that the entity words are related at this first step, a second step is performed to determine whether the calculated entity word relevance score is greater than a predetermined quantitative entity threshold level, as shown at step 624. In one embodiment, the threshold level may be an empirically derived number that indicates a current evaluation of a probability estimate for an identified entity word over a previous observation period. The observation period in one embodiment is six months after the time period, although this time period may vary based on empirical determinations in alternative embodiments. If the following confirmation is obtained: the process ends with a probability relevance score calculated for each entity word in the received search query that is greater than the applied entity word threshold. In contrast, since word relevance is primarily determined based on the presence of (a) the stored domain URL or domain title of the search engine results page and (b) the query word in the received search query, if an entity word has been identified from the parsing of the search engine results page, but is not present in the search query, as shown at step 622, determination of the entity word relevance score is not performed because such word is not considered a "relevant" entity word in this process. In one embodiment, if the calculated entity word relevance score is below the applied entity word threshold level, the word will not be considered a "related" entity word.

FIG. 7A illustrates an embodiment of a process for determining a relevant intent word. The process 700 also begins with the receipt of a user search query, as shown at step 702, followed by the retrieval of a list of URLs associated with the received search query, as shown at step 704. Both the received search query and the retrieved URL associated with the search query are stored in the aggregated search log as part of a process of maintaining a running and dynamically updated log over an adjustable tracking period. Once retrieved, the words in the received query are parsed, as shown at step 706, and each parsed word in the URL associated with the received search query is compared, as shown at step 708. During the parsing process, each query word determined to be in the associated URL list is identified as an "intent" word and added to the itemized list of such words, as shown at step 714, while each query word not in the associated URL list is considered not an intent word, as shown at step 714, and no further action is performed on them.

In FIG. 7B, the process 700 is expanded in one embodiment to illustrate the determination of the intention word relevance score. At this stage of process 700, an analysis of the frequency of occurrence is performed to determine the number of times the identified intent word on the itemized list appears in the URL and received user search query, as shown at step 716. This step is followed by determining the number of times each word in the received search query appears in the received search query and associated list of URLs, as shown at step 718. An intent word relevance score is then calculated for each intent word on the itemized list based on the ratio of the numbers determined in steps 716 and 718. The number generated from this ratio is the intention word relevance score and it represents an optimization of the probability value based on the assumed probability distribution. The number is then compared to a quantitative intent threshold level and if the score is greater than the empirically determined quantitative intent threshold level, it is determined to be a "related" intent word, as shown at step 720.

FIG. 8 is a flow diagram illustrating an embodiment of a process for query set identification. The process 800 begins with the retrieval of related entity words associated with a search query, as shown at step 802, followed by the retrieval of related intent words also associated with the search query, as shown at step 804. The process 800 continues with searching all queries including related entity words and related intent words in the adjustable tracking period of queries and related search engine results pages stored in the aggregated search log, as shown at step 806. In general, the relevance scores computed for the relevant intent words and each of the relevant entity words are used to identify queries to identify and retrieve past user queries that include these relevant entity words or combinations of relevant intent words. The combination of these past search queries comprises a set of queries and is compiled for relative relevance determination, as shown at step 808. This subset of queries, referred to as the "query set," is then stored in the aggregated search log, as shown at step 810, for later retrieval to determine a relative relevance rank.

FIG. 9 illustrates an embodiment of a process for generating a set of sequentially ranked sets of relevance of query suggestions. The process 900 begins with the retrieval of the entity word relevance scores for the associated entity words, as shown at step 902, followed by the retrieval of the intention word relevance score for each of the associated intention words, as shown at step 904. Each pair of an entity word relevance score and an intention word relevance score is an indicator of the relative "importance" of a word in a search query and its relevance. Depending on the calculated entity word relevance scores and intent word relevance scores, one or more past user queries that will include a set of queries associated with the received search query may be retrieved from the aggregated search log, as shown at step 906. After the set of queries is identified and retrieved, a relevance process is performed that compares each relative relevance score such that the most relevant queries in the set of queries are listed from most relevant to least relevant, in one embodiment based on these relevance scores for both the intent and the entity word. The query set is then ranked in order as a query suggestion set based on the entity word relevance score or the intent word relevance score, as shown at step 908, and this sequentially ranked query suggestion set is then transmitted to the client device from which the user query was received for display as an alternative or query suggestion that is sequentially ranked based on the relevance of the originally received user search query.

It should be understood that the various techniques described herein may be implemented in connection with hardware or software or, where appropriate, with both. Thus, the systems and methods of the presently disclosed subject matter, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, cd roms, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the presently disclosed subject matter.

Although an exemplary implementation may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more stand-alone computer systems, the subject matter is not so limited, but may be implemented in connection with any computing environment, such as a network or distributed computing environment. Still further, aspects of the presently disclosed subject matter may be implemented in or across a plurality of processing chips or devices, and storage may similarly be effected across a plurality of devices. Such devices may include, for example, personal computers, network servers, application servers, mobile devices, and handheld devices.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.