System for structured demonstrations of asynchronous collaboration and machine-based arbitration
阅读说明：本技术 用于异步协作和基于机器的仲裁的结构化论证的系统 (System for structured demonstrations of asynchronous collaboration and machine-based arbitration ) 是由 奈吉尔·D·斯特普 大卫·J·休柏 采青·卢 于 2019-12-20 设计创作，主要内容包括：一种用于收集和处理用户输入的方法。在一些实施例中,所述方法包含：向第一用户呈现用于引发第一响应的提示,所述第一响应包含包括一个或多个数字的数字部分,以及解释性部分；从所述第一用户接收所述第一响应；从多个其它用户中的每一个接收多个其它响应中的相应响应；以及向所述第一用户显示其它响应的有序列表。在所述有序列表内,所述多个其它响应中的第二响应可以早于所述多个其它响应中的第三响应,所述第二响应根据距离的量度比所述第三响应距所述第一响应更远。(A method for collecting and processing user input. In some embodiments, the method comprises: presenting to a first user a prompt for eliciting a first response, the first response comprising a numeric portion comprising one or more digits, and an explanatory portion; receiving the first response from the first user; receiving a respective response of a plurality of other responses from each of a plurality of other users; and displaying an ordered list of other responses to the first user. Within the ordered list, a second response of the plurality of other responses may be earlier than a third response of the plurality of other responses, the second response being further from the first response than the third response according to a measure of distance.)
1. A method for collecting and processing user input, the method comprising:
presenting, by processing circuitry, a prompt to a first user for eliciting a first response, the first response comprising:
a number part comprising one or more numbers, an
An explanatory part;
receiving, by the processing circuit, the first response from the first user;
receiving, by the processing circuit, a respective response of a plurality of other responses from each of a plurality of other users;
displaying, by the processing circuitry, an ordered list of other responses to the first user;
determining, by the processing circuit, whether the first user and the other user have agreed;
generating, by the processing circuitry, a consensus forecast in response to determining that the first user and the other users have formed the consensus forecast;
generating, by the processing circuitry, a coordinated forecast in response to determining that the first user and the other users have not formed consensus forecasts; and
causing, by the processing circuitry, an action to be taken in response to the consensus forecast or in response to the coordinated forecast,
wherein:
a second response of the plurality of other responses is earlier in the ordered list than a third response of the plurality of other responses, and
the second response is further from the first response than the third response by a measure of distance.
2. The method of claim 1, further comprising denying the first user access to all other responses until the first response is received from the first user.
3. The method of claim 1, wherein the prompt to elicit the first response comprises a prompt to elicit the digital portion and a prompt to elicit the explanatory portion,
the prompt for causing the digital portion comprises a prompt for causing a forecast; and is
The prompt for eliciting the explanatory portion comprises a prompt for eliciting underlying theory for the forecast.
4. The method of claim 3, wherein the prompt to cause the explanatory portion further comprises a prompt to cause a citation document that supports the underlying theory.
5. The method of claim 1, further comprising presenting a prompt to the first user for causing a rating of the second response.
6. The method of claim 1, further comprising:
displaying the first response to a second user of the plurality of other users, an
Causing a rating of the first response from the second user.
7. The method of claim 6, further comprising presenting a prompt to the second user for causing a rating of the reference file in the first response.
8. The method of claim 7, wherein the prompt for causing the rating of the cited document in the first response comprises:
a prompt for causing an evaluation of relevance of the cited document; and
a prompt for causing an evaluation of the authenticity of the cited document.
9. The method of claim 1, further comprising, after displaying the ordered list of other responses to the first user,
presenting a prompt to the first user to cause adjustment of the first response.
10. The method of claim 1, wherein:
the numerical portion represents an ordinal forecast and is represented by a numerical vector; and is
The measure of distance is a ratio of the first vector and the second vector:
the difference between:
the answer density of the first vector, and
(ii) a density of answers to the second vector; and
a measure of similarity between the first vector and the second vector.
11. The method of claim 10, wherein the answer density for each of the first and second vectors is a weighted sum over elements of the vector, wherein each weight is an index of the respective element.
12. The method of claim 11, wherein the measure of similarity is a ratio of:
dot product of the vectors, and
the product of the magnitudes of the vectors.
13. The method of claim 1, wherein:
the numerical portion represents the classification forecast, and is represented by a numerical vector,
a measure of the distance of a test vector from a reference vector is a ratio of:
the difference between:
the value of the largest element of the reference vector, and
a value of an element of the test vector corresponding to the largest element of the reference vector; and
a measure of similarity between the vectors.
14. The method of claim 13, wherein the measure of similarity is a ratio of:
dot product of the vectors, and
the product of the magnitudes of the vectors.
15. The method of claim 1, wherein:
the numerical portion represents the classification forecast, and is represented by a numerical vector,
a measure of the distance of a test vector from a reference vector is the product of:
the sign term, and
a preliminary distance measure of the test vector from the reference vector.
16. The method of claim 15, wherein the preliminary distance measure of the test vector from the reference vector is an absolute value of a ratio of:
the difference between:
the value of the largest element of the reference vector, and
a value of an element of the test vector corresponding to the largest element of the reference vector; and
the ratio between:
dot product of the vectors, and
the product of the magnitudes of the vectors.
17. The method of claim 16, wherein the sign term is:
is one when the value of the element of the test vector corresponding to the largest element of the reference vector is the largest element of the test vector, and
otherwise it is a negative one.
18. The method of claim 1, further comprising assigning, by a processing circuit, a rating to the first response and to each of the other responses.
19. The method of claim 1, further comprising displaying to the first user a position of the number portion within a range of other responsive number portions.
20. A system for collecting and processing user input, the system comprising:
processing circuit, and
a memory coupled to the processing circuit and having stored thereon instructions that, when executed by the processing circuit, cause the processing circuit to:
presenting, to a first user, a prompt to elicit a first response, the first response comprising:
a number part comprising one or more numbers, an
An explanatory part;
receiving the first response from the first user;
receiving a respective response of a plurality of other responses from each of a plurality of other users; and
displaying an ordered list of other responses to the first user
Determining whether the first user and the other users have agreed;
in response to determining that the first user and the other users have formed a consensus forecast, generating the consensus forecast;
generating a coordinated forecast in response to determining that the first user and the other users have not formed consensus forecasts; and
causing an action to be taken in response to the consensus forecast or in response to the coordinated forecast,
wherein:
a second response of the plurality of other responses is earlier in the ordered list than a third response of the plurality of other responses, and
the second response is further from the first response than the third response by a measure of distance.
Technical Field
One or more aspects in accordance with embodiments of the present disclosure relate to forecast aggregation and arbitration.
Background
Prior art online message boards are sometimes used by users of such message boards to request suggestions or opinions that may be helpful to such users. For example, a user who may want to purchase or sell a certain item may request an opinion regarding the possible future price of the item. The user may then analyze and aggregate the received responses in an ad hoc manner, striving to evaluate the expertise of those who issue the responses, and extract forecasts of future prices from the responses.
This approach may have various disadvantages. On the message board, the user may start a discussion in a "thread," and other users may comment on the original post or posts made by others in response to the original post. This type of discussion may lack any type of discernable structure, although sometimes responses to posts are displayed as a tree for contextual understanding. Regardless of how the structure is handled, such interfaces are of little use in guiding the discussion, which may be left entirely to the participants. Thus, discussions may become disoriented due to a departure, a failure to refer to a source to support a given opinion, or a reference to a source that does not support the argument being made.
Another potential disadvantage relates to the participants themselves. The user may be a victim of his bias as well as his initial arguments and opinions. For example, a user may be affected by a class of bias called "anchor," where the user may be strongly affected by the first external opinion she or he sees or hears. Another potential bias is referred to as "validation bias," which is a type of logical failure in which a person creates an opinion and then verifies it by merely selecting a source and other verification methods that are consistent with it, ignoring any opinions or resources that may be objectionable.
Furthermore, it may be difficult or impossible for participants to achieve any type of consensus, even when building a consensus and drawing a conclusion that is the primary goal of the discussion.
Accordingly, there is a need for systems and methods for facilitating fruitful interactions between online users.
Disclosure of Invention
According to an embodiment of the present invention, there is provided a method for collecting and processing user input, the method comprising: presenting, by processing circuitry, a prompt to a first user for eliciting a first response, the first response comprising: a number portion comprising one or more numbers, and an explanatory portion; receiving, by the processing circuit, the first response from the first user; receiving, by the processing circuit, a respective response of a plurality of other responses from each of a plurality of other users; displaying, by the processing circuitry, an ordered list of other responses to the first user; determining, by the processing circuit, whether the first user and the other user have agreed; generating, by the processing circuitry, a consensus forecast in response to determining that the first user and the other users have formed the consensus forecast; generating, by the processing circuitry, a coordinated forecast in response to determining that the first user and the other users have not formed consensus forecasts; and causing, by the processing circuitry, an action to be taken in response to the consensus forecast or in response to the coordinated forecast, wherein: a second response of the plurality of other responses is earlier in the ordered list than a third response of the plurality of other responses, and the second response is further from the first response than the third response according to a measure of distance.
In some embodiments, the method also includes denying the first user access to all other responses until the first response is received from the first user.
In some embodiments, the prompt for eliciting the first response includes a prompt for eliciting the digital portion and a prompt for eliciting the explanatory portion, the prompt for eliciting the digital portion includes a prompt for eliciting a forecast; and the prompt for eliciting the explanatory portion comprises a prompt for eliciting underlying theory for the forecast.
In some embodiments, the prompt to elicit the explanatory portion further comprises a prompt to elicit a citation document that supports the underlying theory.
In some embodiments, the method further includes presenting a prompt to the first user for causing a rating of the second response.
In some embodiments, the method further comprises: displaying the first response to a second user of the plurality of other users, and eliciting a rating of the first response from the second user.
In some embodiments, the method further includes presenting a prompt to the second user for causing a rating of the reference file in the first response.
In some embodiments, the prompt for causing the rating of the cited document in the first response comprises: a prompt for causing an evaluation of relevance of the cited document; and a prompt for causing an evaluation of the authenticity of the cited document.
In some embodiments, the method also includes presenting a prompt to the first user for causing adjustment of the first response after displaying the ordered list of other responses to the first user.
In some embodiments, the numerical portion represents an ordinal forecast and is represented by a numerical vector; and the measure of distance is a ratio of the first vector and the second vector: a difference between the answer density of the first vector and the answer density of the second vector; and a measure of similarity between the first vector and the second vector.
In some embodiments, the answer density for each of the first and second vectors is a weighted sum over elements of the vector, where each weight is an index of the respective element.
In some embodiments, the measure of similarity is a ratio of: a dot product of the vectors, and a product of magnitudes of the vectors.
In some embodiments, the numerical portion represents a classification forecast and is represented by a numerical vector, the measure of the distance of a test vector from a reference vector being a ratio of: a difference between a value of a largest element of the reference vector and a value of an element of the test vector corresponding to the largest element of the reference vector; and a measure of similarity between the vectors.
In some embodiments, the measure of similarity is a ratio of: a dot product of the vectors, and a product of magnitudes of the vectors.
In some embodiments, the numerical portion represents a classification forecast and is represented by a numerical vector, the measure of the distance of a test vector from a reference vector being the product of: a sign term, and a preliminary distance measure of the test vector from the reference vector.
In some embodiments, the preliminary distance measure of the test vector from the reference vector is an absolute value of a ratio of: a difference between a value of a largest element of the reference vector and a value of an element of the test vector corresponding to the largest element of the reference vector; and a ratio between the dot product of the vector and the product of the magnitudes of the vector.
In some embodiments, the sign term is: one when the value of the element of the test vector corresponding to the largest element of the reference vector is the largest element of the test vector, and negative one otherwise.
In some embodiments, the method also includes assigning, by the processing circuit, a rating to the first response and to each of the other responses.
In some embodiments, the method also includes displaying to the first user a position of the numeric portion within a range of other responsive numeric portions.
According to an embodiment of the present invention, there is provided a system for collecting and processing user input, the system comprising: a processing circuit, and a memory coupled to the processing circuit and having stored thereon instructions that, when executed by the processing circuit, cause the processing circuit to: presenting, to a first user, a prompt for eliciting a first response, the first response comprising: a number portion comprising one or more numbers, and an explanatory portion; receiving the first response from the first user; receiving a respective response of a plurality of other responses from each of a plurality of other users; and displaying an ordered list of other responses to the first user, determining whether the first user and the other users have agreed upon; in response to determining that the first user and the other users have formed a consensus forecast, generating the consensus forecast; generating a coordinated forecast in response to determining that the first user and the other users have not formed consensus forecasts; and causing an action to be taken in response to the consensus forecast or in response to the coordinated forecast, wherein: a second response of the plurality of other responses is earlier in the ordered list than a third response of the plurality of other responses, and the second response is further from the first response than the third response according to a measure of distance.
Drawings
These and other features and advantages of the present disclosure will be understood and appreciated with reference to the specification, claims, and appended drawings, wherein:
FIG. 1 is a system block diagram according to an embodiment of the present disclosure;
FIG. 2 is a prompt according to an embodiment of the present disclosure;
FIG. 3 is a prompt according to an embodiment of the present disclosure;
FIG. 4 is a prompt according to an embodiment of the present disclosure;
FIG. 5A is a prompt according to an embodiment of the present disclosure;
FIG. 5B is a scatter plot according to an embodiment of the present disclosure
FIG. 6 is a block diagram illustrating various computerized systems in communication with each other that may be used to implement embodiments of the invention; and
fig. 7 is a block diagram illustrating a processing system, processing circuit, or portion thereof, for use in connection with at least one embodiment of the invention.
Detailed Description
The detailed description set forth below in connection with the appended drawings is intended as a description of exemplary embodiments of a system for structured demonstrations of asynchronous collaboration and machine-based arbitration provided in accordance with the present disclosure and is not intended to represent the only form in which the present disclosure may be constructed or utilized. The description sets forth the features of the disclosure in connection with the illustrated embodiments. It is to be understood, however, that the same or equivalent functions and structures may be accomplished by different embodiments that are also intended to be encompassed within the scope of the disclosure. As indicated elsewhere herein, like element numbers are intended to indicate like elements or features.
FIG. 1 illustrates a system diagram of a system for demonstration in some embodiments. The system for demonstration may be configured by a user who wishes to obtain a series of opinions (or "forecasts") or consensus opinions from multiple users, referred to herein as "hosts". The host may wish to investigate opinions of other users about the questions 105; as an illustrative example, the question 105 may be an estimate (or forecast) of the amount of oil that will be produced by Libya in 5 months of 2020. The host may configure the system according to the question to be answered, and each participating user may then be presented with a prompt at 110 that elicits a response to the question. Fig. 2 shows an example of a prompt to elicit a response. The user is presented with a plurality of sliders 205 and text boxes 210. The slider elicits a numeric portion of the response 215 (the user's forecast of possible oil output, as discussed in further detail below), and the text box elicits an explanatory portion of the response 220 (the basic theory of the user's forecast).
Each of the plurality of sliders 205 corresponds to a range or "answer interval" (a first range being a value less than 750, a second range being a value between 750 and 900, etc.). The user may adjust each slider to reflect a confidence that the user's answer is in the range corresponding to the slider. For example, if the user determines that the amount will be greater than 900 buckets (BBL) and less than 1050 buckets, the user may slide the third slider all the way to the right (indicating that the answer is at 100% confidence in this range). If, from the user's perspective, the amount determination is greater than 750 buckets and less than 1050 buckets, and equally likely (i) less than 900 buckets or (ii) more than 900 buckets, the user may slide each of the second and third sliders half way to the right, i.e., to 50%.
Software (e.g., JavaScript running in a browser that the user interacts with the system for demonstration) may automatically adjust the slider so that the total of the indicated four values is 100%. The algorithm may prioritize the slider that the user has not adjusted to avoid having the slider setting change affect the previously made setting of another slider, which is not desired by the user. For example, in some embodiments, if the user had adjusted the first slider to 50%, and then attempted to adjust the second slider beyond 50% (resulting in a total of > 100%), the system would simply "card" the value of the second slider to the highest allowable value for the second slider such that its sum totals 100%. In some embodiments, once the first slider is set, the system will not change it, if there is no explicit user intervention, which may involve the user manually changing the slider, or the user clicking a button that normalizes all sliders to have equal values (50% for 2 sliders, 25% for 4 sliders, etc.).
In text box 210, the user may provide basic theories for forecasting. The underlying theory may contain explanatory words entered by the user. Furthermore, because the argument may be futile unless a participant can reference a related and real source to support their forecast, the system may provide a method for users to reference a source that supports their point of regard, such as by inserting a URL in their underlying theory, providing the URL alone in the "reference file upload" column, or uploading a Document in a network Format such as Portable Document Format (pdf). These reference files may then be shown to other participants as internet links or downloadable document files, which other participants may examine and include in their arguments, for example by demonstrating a particular point of the reference file or providing their own corresponding reference file. Other users may also evaluate the citation file (as discussed in further detail below).
The user may be presented with the prompt of fig. 2 without prior or concurrent access to any forecasts that other users may have made (i.e., the user may be denied such access until a response to the prompt is received from the user); this feature (the forecast of denying users access to other users) may eliminate anchoring and mitigate validation bias, e.g., reduce validation bias. The validation bias may also be mitigated by selecting the order in which the responses of the other users are later displayed (as discussed in further detail below).
The hints of FIG. 2 trigger a type of forecasting that may be referred to as "ordinal forecasting". As used herein, "ordinal forecasts" are those forecasts in which there is ordering of possible answer intervals, such as the case of the intervals in fig. 2 (each corresponding to a respective slider). A forecast in which no such ordering exists may be referred to as a "classification forecast". An example of a classification forecast is a prediction in which, for example, four candidates will win an election. In this case, the hint may be similar to FIG. 2, with a first slider labeled with the name of the first candidate (instead of "less than 750" as in FIG. 2), a second slider labeled with the name of the second candidate, and so on. In this forecasting, the user may also use a slider to indicate the degree of confidence, e.g., adjust the second slider all the way to the right if the user determines that the second candidate will win. Whether the forecast is an ordinal or a classification can affect how the "distance" between the forecasts of two users (a measure of the degree to which the two users are inconsistent) can be calculated (as discussed in further detail below). The ordinal forecast and the classification forecast may each be represented by a vector (e.g., a vector having one element for each range or answer region or, in the case of classification forecast, for each item (e.g., candidate)), each element of the vector representing a degree of user confidence that the corresponding range or item is a correct answer to a question posed by the host.
It should be understood that instead of a slider, any other interface suitable for receiving numeric input from a user may be used (e.g., a set of text boxes, each for receiving a respective number), the number of input fields or controls may be different than the four shown in FIG. 2, and the controls may be constrained or automatically adjusted to have some total (e.g., 100%) of features that may not be present (in which case the system may normalize the vector of numeric input values before further processing, as appropriate). Referring again to FIG. 1, after responses have been elicited (at 110) from multiple users, the forecasts can be ranked (at 115) and displayed (at 120) to each user after the user has provided an initial response (e.g., by responding to the prompts of FIG. 2).
Other forecasts displayed to any user may be sorted (or "sorted" or "ranked") (at 115, fig. 1), with the first displayed (or "earliest") response being the forecast that is least consistent with (i.e., furthest from) the user's forecast. This may enable the user to review and contrast conflicting opinions; in this way, the system for demonstration can mitigate validation bias and drive the population users to self-evaluate and constructive dispute. FIG. 3 shows an example of how forecasts can be displayed to a particular user (or "current user"). Summary graph 305 may show points for each of a plurality of other users 'forecasts, where the horizontal position of each point shows the distance of the other user's forecasts from the current user's forecast, and the vertical position of each point shows the rating of each other user's forecast (by the other users, discussed in further detail below). The vertical line may show a horizontal position corresponding to the forecast for the current user. The "distance" may be a signed amount, as discussed in further detail below.
Below the summary graph 305, two lists are displayed that may show the forecasts of other users, with (as mentioned above) the forecasts that are most different from the current user's forecast listed first, i.e., at the top of each list. The left-hand list 310 in fig. 3 shows forecasts with negative distances from the current user's forecast (other forecasts with the greatest (negative) distances from the current user's forecast are shown first), and the right-hand list 315 on the right in fig. 3 shows forecasts with positive distances from the current user's forecast (other forecasts with the greatest (positive) distances from the current user's forecast are shown first). Because some users may be accustomed to being able to change the order of the displayed lists from a default order, the system may give each user the option of searching for and re-ordering each list.
The forecasted answer intervals for each other user may be shown as a color bar 320 at the top of each forecast (where the color in the bar corresponds to one of the answer intervals and the length of the segment with that color corresponds to the confidence value for that answer interval). The rating bar 325 may show the number of "approved" tickets received for the forecast. In some embodiments, the prompt of FIG. 2 remains displayed when the prompt of FIG. 3 is shown (the prompt of FIG. 2 may be shown, for example, directly above the prompt of FIG. 3) or when the prompt of FIG. 4 is shown (discussed below). This may make it possible for the current user to adjust her or his response to the host problem, which has been influenced, for example, by some other user's underlying theory.
The system for demonstration may elicit (at 125, FIG. 1) from each user an assessment of the underlying theory provided by the other users. When the current user hovers over (e.g., places a mouse pointer over) any other user's forecasts, the current user may be presented with an opinion rating prompt, an example of which is shown in FIG. 4. The opinion rating prompt 405 may give the user the opportunity to cast a "favorable" ticket by "answering a" yes "to the basic theory that the question is" good "or cast a" negative "ticket by answering a" no ". Further, in the embodiment of FIG. 4, a small speaking bubble icon 410 is a button that can open an edit bar to add comments to the underlying theory. The actual text edit bar may appear similar to a box, within the boundaries of the opinion rating prompt 405 and underneath the text of the opinion rating prompt 405. The addition of textual comments may be to "is this a good basic theory? "yes/no answer to question supplement. In addition to providing visible low-effort feedback that may encourage participants to participate in the system more often, allowing users to provide an overall assessment of a given point of regard also provides metrics that may be used to rank and aggregate underlying theories.
To help track the values of user-provided references, the system for demonstration may elicit (at 130, FIG. 1) from each user an evaluation of other user-provided references. It may be advantageous to evaluate each reference file at least on measures of relevance and authenticity, since these are orthogonal pieces of information for a given reference file; one reflects the relationship between the source (e.g., a referenced document or publication) and the argument, and the other reflects the value of the source itself. For example, a given source may be accurate and trusted, but a user relying on the cited document may have misinterpreted its meaning, causing it to be incorrectly used to support statements that do not follow the information in the cited source. Rather, participants may refer to sources that fully support the theory, but the referenced sources may be broadly construed as untrusted or highly biased; in this case too, the utility of the cited document in question may be low. In particular, the source of multiple negative tickets received from the subject matter expert in terms of authenticity may be removed completely from the discussion. By eliciting feedback on relevance and authenticity, the system allows individual arguments to be weighted and judged, and facilitates fruitful discussions by encouraging the use of external sources rather than purely opinion-based arguments.
The system for demonstrations may also induce (at 135, fig. 1) user comments on the comments. However, the depth of the discussion tree may be limited to such comments (e.g., the system may make it impossible for the user to provide comments on the comments of the underlying theoretical comments) to reduce the risk of discussion divulgence.
When the current user hovers over any reference file in another user's forecast, the current user may be presented with a prompt for causing a rating of the reference file, an example of which is shown in FIG. 5A. The hint of FIG. 5A includes: a relevance slider 505 that the current user can use to indicate the relevance of the referenced files; and an authenticity slider 510 that the current user can use to indicate the authenticity of the referenced document. The aggregated two-dimensional reference file evaluation may then be displayed (e.g., to the current user, or to the host), for example, in the scatter plot of FIG. 5B. Each reference file has a corresponding evaluation set and a corresponding scatter plot, such as that of fig. 5B. In some embodiments, users are categorized by various characteristics (e.g., by the degree of expertise, as shown in FIG. 5B, or according to one or more other characteristics, such as whether the user is a teammate of the person supplying the cited documents). The classification may be indicated in the scatter plot by, for example, displaying each point in a corresponding color.
The distance between forecasts can be defined and calculated as follows. For ordinal forecasting, the forecast can be represented as a probabilistic quality function p over N answer intervals, such that each answer pi∈[0，1]And isThe forecasted answer density may be a weighted sum of probabilities, where the weights are based on the answer interval, e.g., each weight may be a midpoint of the respective interval or an index of the interval. For example, answer density may be defined as
The similarity between the two forecasts p and q is taken as its correlation,
and the forecast distance is then defined as
If p is a forecast for a particular user, this metric is positive for forecasts that are higher weighted (i.e., have higher answer densities) than p and negative for forecasts that are lower weighted (i.e., have lower answer densities) than p, scaled by similarity in both cases. This allows for meaningful segmentation of the forecast collection into
F+＝{q|Dord(p, q) ≥ 0} and F-＝{q|Dord(p，q)＜0}
For classification forecasting, distance may be defined in either of two ways. The first definition, which may be referred to as "agreement on maximum answer value", relates to the extent to which two forecasts agree on a first choice of one of the forecasts. That is, if user 1 assigns the maximum value to candidate B, then all other forecasts are compared by their candidate B values. The second option, which may be referred to as "agreement on maximum interval index", first relates to whether the two forecasts select the same interval as their maximum. The details of these two definitions are provided below. In each case, p is the "reference" forecast, which is compared to the "test" forecast q.
To calculate the distance based on agreement with the maximum answer value, the system may first select the interval containing the maximum value of the reference forecast,
m＝argmax{p}
the system may then calculate the distance over the interval to forecast similarity scaling:
where the forecast similarity r (p, q) is defined as in the case of ordinal forecast.
If the forecast q assigns a large value to the interval m, this metric is positive, which is interpreted as more extreme agreement on the forecast p. If the forecast q assigns a small value to the interval m, this metric is negative, which is interpreted as less extreme agreement on the forecast p.
To calculate the distance with agreement to the maximum interval index, the system may select the maximum interval index for each forecast:
m＝argmax{p}
n＝argmax{q}
the metric may be chosen to be positive if the indices are equal, and negative otherwise, in the absolute value of a previously defined distance metricScaling (used as a preliminary distance measure in the equation below):
wherein the visible symbolic item-1sgn|m-n|One when the value of the element of the test vector corresponding to the largest element of the reference vector is the largest element of the test vector, and negative one otherwise. This metric provides different agreed perspectives, such that the forecast on the positive side of the partition is consistent as to which interval should be assigned the maximum value, and the forecast on the negative side picks up some other interval for the maximum value.
In some embodiments, in addition to allowing users to interactively evaluate each other's responses and references through direct comments, the system automatically evaluates (at 140, FIG. 1) the basic theoretical content and quality of each user through a series of linguistic analysis algorithms. This approach may be used to assign a score to each underlying theory based on the estimated accuracy of the forecasts associated with the underlying theory, providing an alternative or parallel approach to the determination of user-provided votes. Linguistic analysis can handle basic theories and provide a set of features to estimate the accuracy of the forecast. In some embodiments, the extracted features are described in the' 397 application. For example, various sentence structures, word usage, citation file patterns, and the user's past prediction history may be correlated to a training supervised machine learning algorithm (e.g., support vector machine, bayesian network, or deep learning) to calculate a score that is the likelihood that the prediction will be demonstrated to be correct. This can help remove bias from the vote and fill in quality information in instances where there are few user votes.
This automated evaluation may be implemented using a probabilistic graphical model, referred to as probabilistic matrix factorization (PMF-RF) for rationalizing the forecast, which learns the relationships between potential user profiles and the points of talking made in each forecast rationale. This method is described in the' 397 application. Some embodiments of the current system for demonstration may be arranged to implement a PMF-RF model, so that it is able to identify quality ground theories with a high likelihood of accuracy. In some embodiments, the following models are constructed: (1) the forecasting ability of each user (i.e., the user is ultimately demonstrated the correct frequency), (2) the difficulty of a particular problem, and (3) discussions that occur in response to the underlying theory provided by the user. By understanding the relationship between these three variables, the model predicts how accurately the user will be with respect to each question. In small data settings, this model can learn more about the user through the user's basic theory than just by looking at their forecast accuracy. For example, if a model is able to extract features from underlying theory, less historical data may be needed to make predictions about a given user's forecast. This improved understanding of the user's abilities may then be used to provide a more accurate estimate of which users may be making strong points of discourse in either party.
In some embodiments, automated aggregation and coordination (at 145, FIG. 1) is performed to produce a coordinated solution, for example, when users of the system collaborate on issues fail to reach a consensus. For example, if the change in response at a given time (i.e., the "critical" time at which a decision needs to be made) exceeds a threshold, then this may be determined to be present. If the change in response exceeds a threshold, the system may determine that there is no consent and the system may use the information it already has to make decisions on behalf of the group. Another example may be the convergence of opinions over time; as consensus grows, the change may decrease toward zero (where a zero change would indicate 100% agreement between all system participants). If the decrease in change towards zero pauses for a long period of time specified by the person asking the question, the system can declare a deadlock and make a decision based on the information it already has. If users of the system collaborate on the problem do not reach consensus, the system may attempt to use the feedback provided by the users about each other's basic theory and citation files, as well as their own basic theory evaluation algorithms, to generate consensus opinions discussing the problem without further input from the users. This may be accomplished by a scoring mechanism that looks at a basic set of theories of all parties to the dispute and assigns a value to each option based on the weighted votes from the users at each basic theory, comments about the basic theory, and relevance and authenticity scores of the cited documents about the basic theory. This allows the system to assign a single value score to each of the potential options for the problem, which can then be normalized to a probability assignment.
For example, a multitask modeling framework for task assignment may be used that learns the skills of participants in a crowd and then assigns the skills to tasks based on their requirements and the skills of the available participants. This framework is described, for example, in "crowd-sourced sorting Algorithm for Geopolitical Event Forecasting" (a crowdsourceg triangle Algorithm for Geopolitical Event Forecasting) "by m.rostami et al (RecSys'18, 2018, 10 months 2-7 days, BC, usa), which is incorporated herein by reference. In this approach, rather than using information about the participant's skills for assignment, the system may assign weights to individual system inputs (i.e., assertions, ground truth scores, and citation file scores) using the skill and feature information provided by the framework to produce a final aggregated system response. During the learning and parameter inference phases of this approach, the system estimates the parameters of the system inputs (e.g., how correct the underlying theory provider or voter or reviewer was in the past on this topic area, past values for a given reference source, etc.) based on historical performance. This action may also incorporate authenticity and relevance scores for the cited documents, where good reference arguments have a higher weight than arguments of bad references or arguments that do not have references at all. Subsequently, after learning all the model parameters, the system can incorporate the features of the system inputs into the aggregation process by considering the values of the probabilities as described in the framework in equation 1 of the Rostami publication identified above. Thus, consensus opinions are created that are a weighted aggregation of individual participant opinions weighted by their reference files and general feedback scores. For example, in some embodiments, all forecasts are averaged and the average is used as the system response. In other embodiments, a weighted average may be calculated, as described in the' 397 application.
When the system performs coordination on its inputs to reach a decision, it can generate an audit trail for human evaluation to ensure (1) that the system has not made any errors in its processing, and (2) that all participants can see and understand the decisions that the system makes, but with full system transparency. The audit trail generated by the system may contain the following for each issue of system coordination: each base theory ID, each base theory score, each's citation file and a list of relevance and veracity scores, a discussion thread structure (i.e., the level of each base theory or argument in the thread), and a weight for each input in the decision. A graphical user interface may be implemented to display this information to the user or it may be output as a text file.
As used herein, a first forecast is "farther" from a current user than a second forecast if the absolute value of the distance of the first forecast (of the other user) from the current user's forecast exceeds the absolute value of the distance of the second forecast (of the other user) from the current user's forecast. As used herein, a "forecast" is any quantifiable information unknown to an operator, as it is information relating to one or more events that have not occurred, or for other reasons. Thus, assumptions about the root cause of a failure that has occurred may be referred to as "forecasts," even though they are not predictions about future events. As used herein, the "ratio" of the first amount and the second amount means the first amount divided by the second amount. For example, "the ratio of A and B" means A/B. As used herein, the word "or" is inclusive, such that, for example, "a or B" means any of (i) a, (ii) B, and (iii) a and B.
Some or all of the operations described herein may be performed by one or more processing circuits. For example, the system for demonstration may be hosted on a server that includes processing circuitry, and each user and host may use a user interface displayed by a computer that includes processing circuitry (e.g., in a web browser). The server may perform aggregation of user responses, as well as automated point of care evaluation, and automated aggregation and coordination, as discussed above. The term "processing circuitry" is used herein to mean any combination of hardware, firmware, and software for processing data or digital signals. The processing circuit hardware may include, for example, Application Specific Integrated Circuits (ASICs), general or special purpose Central Processing Units (CPUs), Digital Signal Processors (DSPs), Graphics Processing Units (GPUs), and programmable logic devices such as Field Programmable Gate Arrays (FPGAs). As used herein, in a processing circuit, each function is performed by hardware configured (i.e., hardwired) to perform the function, or by more general purpose hardware (e.g., a CPU) configured to execute instructions stored in a non-transitory storage medium. The processing circuitry may be fabricated on a single Printed Circuit Board (PCB) or distributed over several interconnected PCBs. The processing circuitry may include other processing circuitry; for example, the processing circuitry may comprise two processing circuitry FPGAs and a processing circuitry CPU interconnected on a PCB.
FIG. 6 is a block diagram illustrating various computerized systems in communication with each other that can be used to implement embodiments of the invention.
As shown in fig. 6, a system 800 according to some embodiments of the present disclosure interfaces with a server 801 (e.g., a device to be operated) to perform operations described herein, such as generating reports or sending messages based on predicted status changes. In this example, server 801 may include a social media server or other electronic communication device configured to send alerts to users about events.
For example, system 800 may include processing circuitry configured to run an application that provides the functionality of some embodiments. This application may provide each of a plurality of users (e.g., to a respective web browser operated by each user) with a page implementing a user interface (such as those of fig. 2-5B) that allows each user to provide input as well as view and rank the input of other users. The application may also perform aggregation and arbitration of user responses (as discussed above), and it may take action (e.g., take corrective action) in response to the conclusion drawn (e.g., due to user consensus, or due to arbitration), as discussed in further detail below. In some embodiments, the application may be distributed, e.g., the portion of the application that provides the user interface and receives user input may run on a different server than the server that aggregates the user input and may perform the arbitration. System 800 connects to server 801 via network 802 to send and receive information related to various social media networks of interconnected user accounts (element 804) accessed via mobile and non-mobile devices, non-limiting examples of which include desktop computer 806, laptop computer 808, smart phone 810, and other mobile devices 812. Non-limiting examples of user accounts (element 804) includeUser account anda user account. As can be appreciated by one skilled in the art, a user device is any device that can receive and transmit data via the network 802. Additionally, the user account may be a user account of a social media platform that may or may not be able to receive targeted marketing.
Embodiments of the present invention may also be used as an input to a control system. For example, a control system may be a component of a system that may also operate as a system for diagnosing a root cause of a fault (e.g., in an industrial system, nuclear reactor, flood control system, automobile, or the like). In response to identifying the root cause (e.g., reaching a consensus regarding the root cause), the system may take corrective action to remedy the failure (e.g., automated action to open or close the floodgate, in the case of a flood control system).
In other aspects, the server 801 may be a social networking platform or an ad delivery network to access information via a social networking account (element 804) or automatically provide targeted information and/or advertisements to a display screen on a communication device (elements 806, 808, 810).
Computer and other processing circuitry
Various portions of embodiments of the present invention referring to the use of "processing circuitry" may be implemented with logic gates or any other embodiment of a processing unit or processing circuitry. The term "processing unit" or "processing circuit" is used herein to encompass any combination of hardware, firmware, and software for processing data or digital signals.
Fig. 7 is a block diagram illustrating a processing system, processing circuit, or portion of a processing system or processing circuit, referred to herein as a computer system, used in connection with at least one embodiment of the invention.
An exemplary computer system 1200 according to an embodiment is shown in fig. 7. The example computer system 1200 is configured to perform calculations, procedures, operations, and/or functions associated with a program or algorithm. In one embodiment, certain processes and steps discussed herein are implemented as a series of instructions (e.g., a software program) residing in a computer readable memory unit and executed by one or more processing circuits of exemplary computer system 1200. When executed, the instructions cause the exemplary computer system 1200 to perform certain actions and exhibit certain behaviors, such as those described herein.
Exemplary computer system 1200 may include an address/data bus 1210 configured to communicate information. In addition, one or more data processing units, such as processing circuit 1220, are coupled with address/data bus 1210. The processing circuit 1220 is configured to process information and instructions. In one embodiment, the processing circuit 1220 is a microprocessor. Alternatively, the processing circuit 1220 may be a different type of processor, such as a parallel processor, or a field programmable gate array.
Exemplary computer system 1200 is configured to utilize one or more data storage units. The exemplary computer system 1200 may include a volatile memory unit 1230 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled to the address/data bus 1210, wherein the volatile memory unit 1230 is configured to store information and instructions for the processing circuit 1220. The exemplary computer system 1200 may also include a non-volatile memory unit 1240 (e.g., a read-only memory ("ROM"), a programmable ROM ("PROM"), an erasable programmable ROM ("EPROM"), an electrically erasable programmable ROM "EEPROM"), a flash memory, etc.) coupled to the address/data bus 1210, wherein the non-volatile memory unit 1240 is configured to store static information and instructions for the processing circuit 1220. Alternatively, exemplary computer system 1200 may execute instructions retrieved from an online data store, for example, in "cloud" computing. In an embodiment, exemplary computer system 1200 may also include one or more interfaces, such as interface 1250, coupled to address/data bus 1210. The one or more interfaces are configured to enable exemplary computer system 1200 to interface with other electronic devices and computer systems. The communication interfaces implemented by the one or more interfaces may include wired (e.g., serial cable, modem, network adapter, etc.) and/or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technologies.
In one embodiment, exemplary computer system 1200 may include an input device 1260 coupled to address/data bus 1210, wherein input device 1260 is configured to communicate information and command selections to processing circuit 1220. According to one embodiment, the input device 1260 is an alphanumeric input device, such as a keyboard, that may include alphanumeric and/or function keys. Alternatively, the input device 1260 may be an input device other than an alphanumeric input device. In one embodiment, exemplary computer system 1200 may include a cursor control device 1270 coupled to address/data bus 1210, wherein cursor control device 1270 is configured to communicate user input information and/or command selections to processing circuit 1220. In one embodiment, cursor control device 1270 is implemented using a device such as a mouse, trackball, trackpad, optical tracking device, or touch screen. Nonetheless, in one embodiment, cursor control device 1270 is directed and/or activated via input from input device 1260, for example, in response to the use of special keys and key sequence commands associated with input device 1260. In an alternative embodiment, cursor control device 1270 is configured to be guided or steered by voice commands.
In one embodiment, exemplary computer system 1200 may also include one or more optional computer usable data storage devices, such as storage device 1280, coupled to address/data bus 1210. Storage 1280 is configured to store information and/or computer-executable instructions. In one embodiment, the storage device 1280 is a storage device such as a magnetic or optical disk drive (e.g., a hard disk drive ("HDD"), a floppy disk, a compact disk read only memory ("CD-ROM"), or a digital versatile disk ("DVD")). A display device 1290 is coupled to address/data bus 1210, wherein display device 1290 is configured to display video and/or graphics, according to one embodiment. In an embodiment, display device 1290 may comprise a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and/or graphical images and alphanumeric characters recognizable to a user.
Exemplary computer system 1200 is presented herein as an exemplary computing environment in accordance with an embodiment. However, exemplary computer system 1200 is not strictly limited to computer systems. For example, one embodiment provides that exemplary computer system 1200 represents a type of data processing analysis that may be used in accordance with various embodiments described herein. In addition, other computing systems may also be implemented. Indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in one embodiment, one or more operations of various embodiments of the present technology are controlled or implemented using computer-executable instructions, such as program modules, executed by a computer. In an exemplary embodiment, such program modules include routines, programs, objects, components, and/or data structures that are configured to perform particular tasks or implement particular abstract data types. In addition, one embodiment provides for implementing one or more aspects of the present technology through the use of one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or where various program modules are located in both local and remote computer storage media including memory storage devices.
It will be understood that, although the terms "first," "second," "third," etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed herein could be termed a second element, component, region, layer or section without departing from the spirit and scope of the present inventive concept.
Spatially relative terms, such as "below," "lower," "below," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s), as illustrated in the figures. It will be understood that such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the example terms "above" and "below" may encompass both an orientation of above and below. The device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the inventive concepts.
As used herein, the singular forms "a" and "an" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. For example, "at least one of" or the like indicates that the entire list of elements is modified when preceded by a list of elements rather than modifying individual elements of the list. In addition, when describing embodiments of the inventive concept, the use of "may" refer to "one or more embodiments of the present disclosure. Moreover, the term "exemplary" is intended to refer to an example or illustration. As used herein, the term "using" may be considered synonymous with the term "utilizing", respectively.
Any numerical range recited herein is intended to include all sub-ranges subsumed within that range with the same numerical precision. For example, a range of "1.0 to 10.0" is intended to include sub-ranges between (and including) the minimum value of 1.0 and the maximum value of 10.0, that is, having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0, such as 2.4 to 7.6. Any maximum numerical limitation recited herein is intended to include all lower numerical limitations subsumed therein and any minimum numerical limitation recited in this specification is intended to include all higher numerical limitations subsumed therein.
Although exemplary embodiments of a system for structured demonstrations of asynchronous collaboration and machine-based arbitration have been described and illustrated in detail herein, those skilled in the art will appreciate numerous modifications and variations. It is therefore to be understood that a system for structured demonstrations of asynchronous collaboration and machine-based arbitration, constructed according to the principles of the present disclosure, may be implemented in ways other than those specifically described herein. The invention is also defined in the following claims and equivalents thereto.