阅读说明：本技术 建议在线系统中的实体创建内容并向内容添加标签 (Suggesting entities in an online system to create content and add tags to the content ) 是由 史蒂夫·莫林 欧阳雪薇 马丁·姆罗兹 阿努杰·马登 于 2021-04-27 设计创作，主要内容包括：本申请涉及建议在线系统中的实体创建内容并向内容添加标签。在线系统访问基于与一组内容项相关联的主题和该组内容项的内容而训练的模型。在线系统应用该模型来基于多个内容项中的每一个的内容预测该多个内容项中的每一个与主题相关联的概率,并且识别与至少阈值概率相关联的内容项。在线系统检索描述用户对所识别的内容项的参与的信息,并基于该信息确定主题的性能度量的值。如果该值至少是阈值,并且在线系统从实体接收描述与该主题相关联的项目的内容,则在线系统向该实体传送创建描述该项目的内容项的推荐,并且在确定这样做的机会时添加与该项目相关联的标签。(The application relates to suggesting entities in an online system to create content and add tags to the content. The online system accesses a model trained based on topics associated with a set of content items and content of the set of content items. The online system applies the model to predict a probability that each of the plurality of content items is associated with a topic based on the content of each of the plurality of content items, and identifies content items associated with at least a threshold probability. The online system retrieves information describing user engagement with the identified content item and determines a value of a performance metric for the topic based on the information. If the value is at least the threshold value and the presence system receives content from the entity that describes an item associated with the topic, the presence system transmits a recommendation to the entity to create a content item that describes the item and adds a tag associated with the item when determining an opportunity to do so.)

1. A method, comprising:

retrieving a plurality of content items maintained in an online system;

accessing a trained machine learning model that is trained based at least in part on a topic associated with each of a set of content items and content comprising each of the set of content items;

applying the trained machine learning model to predict a probability that each of the plurality of content items is associated with the topic, the probability being predicted based at least in part on content comprising each of the plurality of content items;

identifying one or more of the plurality of content items associated with at least a threshold probability;

retrieving information describing user engagement of the identified one or more content items by one or more users of the online system;

determining a value of a performance metric for the topic based at least in part on the retrieved information;

determining that a value of the performance metric is at least a threshold value;

receiving content describing an item from an entity having presence on the online system, wherein the item is associated with the topic;

determining an opportunity to communicate a recommendation to the entity to create a content item that describes the item and to add a tag associated with the item to the content item; and

transmitting the recommendation to the entity.

2. The method of claim 1, further comprising:

retrieving a set of attributes associated with each of the one or more users; and

determining a set of positioning criteria associated with the topic based at least in part on a set of attributes associated with each of the one or more users.

3. The method of claim 2, wherein the recommendation further comprises one or more options selected from the group consisting of: an option to specify an audience for the content item, and an option to add one or more images of the item to the content item, wherein the audience for the content item corresponds to positioning criteria associated with the topic.

4. The method of claim 3, further comprising:

receiving a request from the entity to create a content item that describes the item, wherein the request specifies that an audience for the content item corresponds to positioning criteria associated with the topic;

generating a content item describing the item based at least in part on the request;

determining an opportunity to present content to a viewing user of the online system;

identifying a set of candidate content items eligible for presentation to the viewing user based at least in part on the positioning criteria, wherein the content items are included among the set of candidate content items;

ranking the content items in the set of candidate content items based at least in part on the positioning criteria;

selecting one or more content items from the set of candidate content items for presentation to the viewing user based at least in part on the ranking; and

transmitting the selected one or more content items for presentation to the viewing user.

5. The method of claim 3, wherein the content describing the item received from the entity comprises a product catalog associated with the entity.

6. The method of claim 5, wherein the one or more images of the item comprise one or more images of a product comprising a product catalog associated with the entity.

7. The method of claim 5, wherein the tag is pre-populated with product-describing information including a product catalog associated with the entity.

8. The method of claim 1, wherein the information describing user engagement of the identified one or more content items by one or more users of the online system comprises one or more selected from the group consisting of: a number of times each of the identified one or more content items is viewed, a number of times each of the identified one or more content items is shared, a number of times each of the identified one or more content items is saved, a number of times one or more users of the online system express preferences for each of the identified one or more content items, and a number of times each of the identified one or more content items is commented.

9. The method of claim 1, wherein the recommendation is transmitted to the entity with a value of a performance metric for the topic.

10. The method of claim 1, wherein the probability that each of the plurality of content items is associated with the topic is predicted based at least in part on one or more selected from the group consisting of: a tag comprising one or more of the plurality of content items, an image comprising one or more of the plurality of content items, and a video comprising one or more of the plurality of content items.

11. A computer program product comprising a computer readable storage medium having instructions encoded thereon, which when executed by a processor, cause the processor to:

retrieving a plurality of content items maintained in an online system;

accessing a trained machine learning model that is trained based at least in part on a topic associated with each of a set of content items and content comprising each of the set of content items;

applying the trained machine learning model to predict a probability that each of the plurality of content items is associated with the topic, the probability being predicted based at least in part on content comprising each of the plurality of content items;

identifying one or more of the plurality of content items associated with at least a threshold probability;

retrieving information describing user engagement of the identified one or more content items by one or more users of the online system;

determining a value of a performance metric for the topic based at least in part on the retrieved information;

determining that a value of the performance metric is at least a threshold value;

receiving content describing an item from an entity having presence on the online system, wherein the item is associated with the topic;

determining an opportunity to communicate a recommendation to the entity to create a content item that describes the item and to add a tag associated with the item to the content item; and

transmitting the recommendation to the entity.

12. The computer program product of claim 11, wherein the computer-readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

retrieving a set of attributes associated with each of the one or more users; and

determining a set of positioning criteria associated with the topic based at least in part on a set of attributes associated with each of the one or more users.

13. The computer program product of claim 12, wherein the recommendation further comprises one or more options selected from the group consisting of: an option to specify an audience for the content item, and an option to add one or more images of the item to the content item, wherein the audience for the content item corresponds to positioning criteria associated with the topic.

14. The computer program product of claim 13, wherein the computer-readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:

receiving a request from the entity to create a content item that describes the item, wherein the request specifies that an audience for the content item corresponds to positioning criteria associated with the topic;

generating a content item describing the item based at least in part on the request;

determining an opportunity to present content to a viewing user of the online system;

identifying a set of candidate content items eligible for presentation to the viewing user based at least in part on the positioning criteria, wherein the content items are included among the set of candidate content items;

ranking the content items in the set of candidate content items based at least in part on the positioning criteria;

selecting one or more content items from the set of candidate content items for presentation to the viewing user based at least in part on the ranking; and

transmitting the selected one or more content items for presentation to the viewing user.

15. The computer program product of claim 13, wherein the content describing the item received from the entity comprises a product catalog associated with the entity.

16. The computer program product of claim 15, wherein the one or more images of the item comprise one or more images of a product comprising a product catalog associated with the entity.

17. The computer program product of claim 15, wherein the tag is pre-populated with information describing a product, the information describing a product comprising a product catalog associated with the entity.

18. The computer program product of claim 11, wherein the information describing user engagement of the identified one or more content items by one or more users of the online system comprises one or more selected from the group consisting of: a number of times each of the identified one or more content items is viewed, a number of times each of the identified one or more content items is shared, a number of times each of the identified one or more content items is saved, a number of times one or more users of the online system express preferences for each of the identified one or more content items, and a number of times each of the identified one or more content items is commented.

19. The computer program product of claim 11, wherein the recommendation is transmitted to the entity with a value of a performance metric for the topic.

20. The computer program product of claim 11, wherein the probability that each of the plurality of content items is associated with the topic is predicted based at least in part on one or more selected from the group consisting of: a tag comprising one or more of the plurality of content items, an image comprising one or more of the plurality of content items, and a video comprising one or more of the plurality of content items.

Technical Field

The present disclosure relates generally to online systems, and more particularly, to suggesting entities (entities) in an online system to create content describing an item associated with a topic (topic) having at least a threshold of a performance metric and add a tag describing the item to the content.

Background

Online systems allow their users to connect and communicate with other online system users. Users create profiles (profiles) in online systems that are associated with their identities and include information about the users (e.g., interests and demographic information). The user may be an individual or an entity (e.g., a company or charity). Due to the popularity of online systems and the large amount of user-specific information maintained in online systems, online systems provide an ideal forum for users to share content by creating content items (content items) for presentation to other online system users. For example, users may share their uploaded photos or videos by creating content items that include photos or videos that are presented to other users in the online system with whom they are associated.

To facilitate distribution of content, online systems typically allow their users to add tags to content items, which facilitate other online system users searching for content items. When creating a content item, an online system user may add a tag highlighting keywords or topics that may be associated with content included in the content item. The tag may consist of a pound (#) followed by text (e.g., keywords, phrases, categories, topics, etc.) or emoticons (emoji), and may be added to a portion of the content item (e.g., in a title, in a review of the content item, etc.). For example, when creating a content item that includes a photograph of a flower, an online system user may add a tag to the content item by including "# flower (# flower)" and "# photograph (# photo)" in the title. In this example, once a content item is created, when other online system users search for a content item that includes one or more tags, the content item may be included among the content items returned in the search results. Because tags provide online system users with a convenient way to search for currently popular or "trending" topics, users who create content items may be more inclined to create content items associated with trending topics and add tags corresponding to the trending topics (trending topics) to their content items.

However, an online system user who potentially may create a content item associated with a topical topic may not know that the topic is topical and may forgo creating the content item or may create the content item but forego adding a tag corresponding to the topical topic. For example, assume that the topic of a wallet is currently topical and that most online system users searching for content items associated with that topic are searching for content items that include the tag "# wallet (# leather). In this example, it is also assumed that the entity having presence in the online system has previously provided a product catalog to the online system, and that the wallet is included in a product included in the product catalog. In the example above, while the entity may create one or more content items associated with a topical topic, the entity may not do so if the entity does not know that the topic is topical. Further, in the above example, even if an entity creates a content item associated with a topical topic, the online system user may not find the content item if the content item does not include the tag "# wallet" (e.g., if the tag is misspelled), or if the content item includes a different tag or combination of tags that the online system user does not often use to search for content.

SUMMARY

To facilitate distribution of content, online systems typically allow their users to add tags to content items when they are created, which facilitates other online system users searching for content items. Because tags provide online system users with a convenient way to search for topical subjects, users who create content items may be more inclined to create content items associated with topical subjects and add tags corresponding to the topical subjects to their content items. However, an online system user who potentially may create a content item associated with a topical topic may not know that the topic is topical and may forgo creating the content item or may create the content item but forego adding a tag corresponding to the topical topic.

To encourage online system users to create content items associated with topical topics and add tags corresponding to the topical topics to the content items, the online system suggests entities in the online system to create content items that describe items associated with topics having at least a threshold value of a performance metric and add tags describing the items to the content items. More specifically, the online system retrieves a plurality of content items maintained in the online system and accesses a machine learning model trained based on topics associated with a set of content items and content of the set of content items. The online system applies the model to predict a probability that each retrieved content item is associated with a topic based on the content of each retrieved content item, and identifies one or more content items associated with at least a threshold probability. The online system retrieves information describing user engagement with the identified content item and determines a performance metric value for the topic based on the information. If the online system determines that the value is at least the threshold value and receives content describing an item associated with the topic from an entity that is present on the online system, the online system determines an opportunity to transmit a recommendation to the entity to create a content item describing the item and to add a tag associated with the item to the content item. The online system then transmits the recommendation to the entity.

Brief Description of Drawings

FIG. 1 is a block diagram of a system environment in which an online system operates, according to an embodiment.

FIG. 2 is a block diagram of an online system according to an embodiment.

FIG. 3 is a flow diagram of a method for suggesting entities in an online system to create content that describes an item associated with a topic having at least a threshold value of a performance metric and add a tag describing the item to the content, according to an embodiment.

FIG. 4 is an example of a user interface including a recommendation to create a content item describing an item and add various components to the content item, according to an embodiment.

The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

Detailed Description

System architecture

FIG. 1 is a block diagram of a system environment 100 for an online system 140. The system environment 100 shown in FIG. 1 includes one or more client devices 110, a network 120, one or more third party systems 130, and an online system 140. In alternative configurations, different and/or additional components may be included in system environment 100.

Client device 110 is one or more computing devices capable of receiving user input as well as sending and/or receiving data via network 120. In one embodiment, client device 110 is a conventional computer system, such as a desktop computer or laptop computer. Alternatively, client device 110 may be a computer-enabled device, such as a Personal Digital Assistant (PDA), mobile phone, smart phone, or other suitable device. Client device 110 is configured to communicate via network 120. In one embodiment, the client device 110 executes an application that allows a user of the client device 110 to interact with the online system 140. For example, the client device 110 executes a browser application to enable interaction between the client device 110 and the online system 140 via the network 120. In another embodiment, the client device 110 is operated by a native operating system (e.g., running on the client device 110)Or ANDROID^TM) To interact with the online system 140.

Client device 110 is configured to communicate via network 120 using a wired and/or wireless communication system, and network 120 may include any combination of local and/or wide area networks. In one embodiment, network 120 uses standard communication technologies and/or protocols. For example, the network 120 includes communication links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), and so forth. Examples of network protocols for communicating via network 120 include multiprotocol label switching (MPLS), transmission control protocol/internet protocol (TCP/IP), hypertext transfer protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged over network 120 may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML). In some embodiments, all or some of the communication links of network 120 may be encrypted using any suitable technique or techniques.

One or more third party systems 130 may be coupled to the network 120 to communicate with the online system 140, as will be further described below in conjunction with FIG. 2. In one embodiment, the third party system 130 is an application provider that communicates information describing an application for execution by the client device 110 or communicates data to the client device 110 for use by an application executing on the client device 110. In other embodiments, the third party system 130 (e.g., content publisher) provides content or other information for presentation through the client device 110. The third party system 130 may also communicate information to the online system 140, such as advertisements, content, or information about applications provided by the third party system 130.

FIG. 2 is a block diagram of the architecture of the presence system 140. The online system 140 shown in FIG. 2 includes a user profile store 205, a content store 210, an action logger 215, an action log 220, an edge store 225, a topic extraction engine 230, a machine learning module 235, a prediction module 240, a performance determination module 245, an audience determination module (audience determination module)250, a user interface generator 255, a content selection module 260, and a web server 265. In other embodiments, the presence system 140 may include additional, fewer, or different components for various applications. Conventional elements (e.g., network interfaces, security functions, load balancers, failover servers, management and network operations consoles, etc.) are not shown so as not to obscure the details of the system architecture.

Each user of the online system 140 is associated with a user profile that is stored in the user profile store 205. The user profile includes declarative information about the user that is explicitly shared by the user, and may also include profile information inferred by the online system 140. In one embodiment, the user profile includes a plurality of data fields, each data field describing one or more attributes of a respective online system user. Examples of information stored in a user profile include biographical information, demographic information, and other types of descriptive information (e.g., work experience, educational history, gender, hobbies or preferences, location, etc.). The user profile may also store other information provided by the user, such as images or videos. In some embodiments, the user's image may be tagged with information identifying the online system user displayed in the image, where the information identifying the user's tagged image is stored in the user profile of the user. The user profiles in the user profile store 205 may also maintain references to actions performed by the respective users on content items in the content store 210 and stored in the action log 220.

While the user profiles in the user profile store 205 are typically associated with individuals, allowing individuals to interact with each other via the online system 140, the user profiles may also be stored for entities such as businesses or organizations. This allows entities to establish presence in the presence system 140 for associating and exchanging content with other presence system users. An entity may use a brand page associated with the entity's user profile to publish information about itself, about its products, or to provide other information to users of the online system 140. Other users of the online system 140 may be affiliated with the brand page to receive information published to or from the brand page. The user profile associated with the brand page may include information about the entity itself to provide the user with background or informational data about the entity.

The content store 210 stores objects that each represent various types of content. Examples of content represented by an object include page posts, status updates, photos, videos, links, shared content items, game application achievements, check-in events at local businesses, pages (e.g., brand pages), advertisements, or any other type of content. The online system user may create objects stored by the content store 210, such as status updates, photos, events, groups, or applications that are tagged by the user as being associated with other objects in the online system 140. In some embodiments, the object is received from a third party application or a third party application independent of the online system 140. In one embodiment, the objects in the content store 210 represent a single piece of content (single piece of content) or a content "item (item)". Thus, online system users are encouraged to communicate with each other by publishing content items of text and various types of media to the online system 140 via various communication channels. This increases the amount of user interaction with each other and increases the frequency with which users interact within the online system 140.

Action recorder 215 receives communications regarding user actions internal and/or external to online system 140, populates action log 220 with information regarding the user actions. Examples of actions include adding a connection to another user, sending a message to another user, uploading an image, reading a message from another user, viewing content associated with another user, and attending an event published by another user. Further, many actions may involve one object and one or more particular users, and thus these actions are also associated with these users and stored in the action log 220.

The online system 140 may use the action log 220 to track user actions in the online system 140, as well as to track actions in third party systems 130 that communicate information to the online system 140. The user may interact with various objects in the online system 140 and information describing these interactions is stored in the action log 220. Examples of interactions with objects include: comment posts, share links, check-in at a physical location via the client device 110, access content items, and any other suitable interaction. Additional examples of interactions with objects in the online system 140 included in the action log 220 include: reviewing photo albums, communicating with users, correlating with objects, joining events, joining groups, creating events, authorizing applications, using applications, expressing preferences for objects ("like") objects, and participating in transactions. Further, the action log 220 may record user interactions with advertisements in the online system 140 as well as interactions with other applications running in the online system 140. In some embodiments, data from the action log 220 is used to infer interests or preferences of the user, enhance the interests included in the user profile of the user, and allow for a more complete understanding of the user preferences.

The action log 220 may also store user actions taken on the third party system 130 (e.g., an external website) and communicated to the online system 140. For example, the e-commerce website may identify the user of the online system 140 by a social plug-in (social plug-in) that enables the e-commerce website to identify the user of the online system 140. Because the user of the online system 140 is uniquely identifiable, the e-commerce website (e.g., in the previous example) may communicate information about the user's actions external to the online system 140 for association with the user. Thus, the action log 220 may record information about actions performed by the user on the third-party system 130, including web browsing history, participating advertisements, completed purchases, and other patterns from shopping and purchases. Further, actions performed by the user via an application associated with the third-party system 130 and running on the client device 110 may be communicated to the action logger 215 for storage by the application in the action log 220 for recording by the online system 140 and association with the user.

In one embodiment, the edge store 225 stores information describing associations between users and other objects in the online system 140 as edges. Certain edges may be defined by the user, allowing the user to specify their relationships with other users. For example, a user may generate edges with other users that are parallel to the user's real-life relationships (e.g., friends, colleagues, buddies, etc.). Other edges are generated when a user interacts with objects in the online system 140 (e.g., expresses interest in pages in the online system 140, shares links with other users of the online system 140, and comments made by other users of the online system 140).

In one embodiment, the edges may include various features, each feature representing a characteristic of an interaction between users, an interaction between a user and an object, or an interaction between objects. For example, features included in edges describe the rate of interaction between two users, how recently two users have interacted with each other, the rate or amount of information one user retrieves about an object, or the number and type of comments a user posts about an object. These features may also represent information describing a particular object or user. For example, the characteristics may represent a user's level of interest in a particular topic, the rate at which the user logs into the online system 140, or information describing demographic information about the user. Each feature may be associated with a source object or user, a target object or user, and a feature value. Features may be specified as expressions based on values describing a source object or user, a target object or user, or interactions between a source object or user and a target object or user; thus, an edge may be represented as one or more feature expressions.

The edge store 225 also stores information about edges, such as affinity scores (affinity scores) of objects, interests, and other users. The online system 140 may calculate an affinity score, or "affinity," over time to approximate the user's interest in another user or object in the online system 140 based on the actions performed by the user. The online system 140 may calculate the affinity of the user over time to approximate the user's interest in another user, object, or topic in the online system 140 based on the actions performed by the user. The calculation of affinity is further described in U.S. patent application No. 12/978,265 filed on 12/23/2010, U.S. patent application No. 13/690,254 filed on 11/30/2012, U.S. patent application No. 13/689,969 filed on 11/30/2012, and U.S. patent application No. 13/690,088 filed on 11/30/2012, each of which is incorporated herein by reference in its entirety. In one embodiment, multiple interactions between a user and a particular object may be stored as a single edge in the edge store 225. Alternatively, each interaction between a user and a particular object is stored as a separate edge. In some embodiments, the associations between users may be stored in the user profile store 205, or the user profile store 205 may access the side store 225 to determine associations between users.

The online system 140 includes a topic extraction engine 230 that identifies one or more topics associated with objects in the content store 210. To identify topics associated with content items, the topic extraction engine 230 identifies anchor words (anchors) included in the content items and determines the meaning of the anchor words, as further described in U.S. patent application No. 13/167,701 filed on 24.6.2011 (U.S. patent No. 9,779,385 issued on 3.10.2017), which is incorporated herein by reference in its entirety. For example, the topic extraction engine 230 determines one or more topics associated with content items maintained in the content store 210. One or more topics associated with a content item are stored and associated with an object identifier corresponding to the content item. In various embodiments, the association between the object identifier and the topic is stored in the topic extraction engine 230 or the content store 210 to simplify retrieval of one or more topics associated with the object identifier or object identifiers associated with a specified topic. Structured information associated with a content item may also be used to extract a topic associated with the content item. The functionality of the topic extraction engine 230 is further described below in conjunction with FIG. 3.

The machine learning module 235 may train a machine learning model to predict a probability that a content item is associated with a topic. Examples of topics to which content items may be associated include people (e.g., private, public characters or groups), political questions or sports, organizations, specific items (e.g., specific products), categories of products or services, book types, dishes (cuisine), diet, and so forth. The machine learning model may be a convolutional neural network, a deep learning model, or any other suitable machine learning model. In some embodiments, the machine learning module 235 may train a plurality of machine learning models that collectively predict the probability that a content item is associated with a topic. The machine learning module 235 may train the machine learning model based on one or more topics associated with the training set of content items and the content included in each training set of content items. Content included in a content item that may be used to train a machine learning model may include various types of structured and/or unstructured data, such as text data, image data, video data, audio data, or any other suitable type of data that may be included in a content item.

The machine learning module 235 may use various techniques to train the machine learning model. In embodiments where the content used to train the machine learning model includes textual data (e.g., text included in a title, comment, tag, metadata, or location associated with the content item), machine learning module 235 may train the machine learning model using one or more natural language processing techniques, such as topic modeling algorithms and/or Word embedding algorithms (e.g., Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (PLSA), Latent Dirichlet Allocation (LDA), Word2Vec, LDA2Vec, etc.). In embodiments where the content used to train the machine learning model includes audio data, the machine learning module 235 may pre-process the audio data, such as by removing noise, transcribing the speech into text (e.g., manually or using audio transcription techniques (e.g., Automatic Speech Recognition (ASR)), etc.). The machine learning module 235 may also train the machine learning model using one or more algorithms that classify the audio data (e.g., into different types of noise, different types of music, etc.). Further, in embodiments where the audio data used to train the machine learning model includes speech transcribed into text, the text may be used to train the machine learning model using one or more natural language processing techniques, as described above. In embodiments where the content used to train the machine learning model includes image data and/or video data, the machine learning module 235 may train the machine learning model using one or more image processing techniques, as described below.

In embodiments where the content used to train the machine learning model includes image data and/or video data, once trained, the machine learning model may first detect one or more objects contained in the content item and then predict the probability that each object is associated with a topic. The machine learning model may detect one or more objects included within an image in the content item (or included within an image in one or more frames of a video included in the content item) by applying one or more object detection methods to the image. The machine learning model may also identify the locations of objects detected in the image (e.g., by generating a bounding box around each object). In various embodiments, the machine learning model uses one or more object detection methods to identify a category or type corresponding to each object detected in the image. For example, using an object detection method, a machine learning model associates different categories or types (e.g., types of products or other items) with objects detected within an image based on the attributes of the objects. The machine learning model then makes one or more predictions, each prediction corresponding to a probability that an object detected in the image is associated with a subject (e.g., a category or type that the machine learning model has associated with the object), and outputs one or more probabilities. In various embodiments, the machine learning model may predict that an object detected in the image corresponds to a particular item (e.g., a product), where the item corresponds to a topic. In such embodiments, the machine learning model may compare the object to an image of the item (e.g., an image of a product included in a product catalog) and make one or more predictions based on the comparison, each prediction corresponding to a probability that the object detected in the image is associated with a subject.

In embodiments where the machine learning model predicts a probability that an object detected within an image included in a content item is associated with a topic, the machine learning module 235 may train the machine learning model based on a training set of images that include images of different objects associated with different topics. In some embodiments, the set of training images may include publicly available information identifying different topics associated with different objects. The machine learning model may also be trained based on attributes (e.g., shapes, edges, curves, etc.) characterizing each training image set and information identifying topics associated with each training image set.

In embodiments where the machine learning model predicts a probability that an object detected within an image corresponds to a particular item, the machine learning module 235 may train the machine learning model based on a comparison of the object detected within the image to the image of the item. In some embodiments, the machine learning module 235 trains the machine learning model to predict the probability that an object detected within an image matches an item based on previous matches of the object detected within the image to different items (e.g., images of different products included in one or more product categories). For example, the machine learning module 235 applies a marker to an object detected in the image based on an attribute of the object (e.g., a logo, trademark, logo, icon, pattern, texture, print, etc.) indicating that the object matches an item (e.g., a product). Based on the attributes of the labels of the objects extracted from the images, the machine learning module 235 trains the machine learning model using any suitable training method or combination of training methods (e.g., using back propagation if the machine learning model is a neural network, curve fitting techniques if the machine learning model is a linear regression model, etc.). The functionality of the machine learning module 235 is further described below in conjunction with fig. 3.

The prediction module 240 retrieves (e.g., as shown in step 305 of fig. 3) the content items maintained in the online system 140 (e.g., from the content store 210). For example, the prediction module 240 accesses the content store 210 and retrieves various content items created by one or more users of the online system 140. In some embodiments, the prediction module 240 may retrieve the content item based on a set of criteria (e.g., based on the creator of the content item, based on when the content item was created, based on a geographic location associated with the content item, etc.). For example, the prediction module 240 may only retrieve content items created by non-merchant users of the online system 140 within the past month. Alternatively, in the above example, the prediction module 240 may only retrieve content items that were created within the past month by the merchant user of the online system 140.

The prediction module 240 also accesses (e.g., as shown in step 310 of fig. 3) the trained machine learning model and applies (e.g., as shown in step 315 of fig. 3) the machine learning model to predict a probability that the content item is associated with the topic. In some embodiments, the machine learning model may be trained by the machine learning module 235, while in other embodiments, the machine learning model may be trained by the third party system 130. To predict the probability that a content item is associated with a topic using a machine learning model, the prediction module 240 provides an input to the machine learning model that includes the content of the content item. As described above, the content of a content item may include various types of structured and/or unstructured data (e.g., text data, image data, video data, audio data, etc.) included in one or more components of the content item (e.g., a title, one or more images, one or more videos, one or more tags, a set of targeting criteria that specifies an audience for the content item, a location, one or more audio files, etc.). Based on the content of the content item, the machine learning model predicts a probability that the content item is associated with the topic. The prediction module 240 then receives an output from the machine learning model corresponding to the prediction probability. In some embodiments, the prediction module 240 may access and apply multiple machine learning models that collectively perform the functions of the machine learning model. The functionality of prediction module 240 is further described below in conjunction with FIG. 3.

The performance determination module 245 identifies (e.g., as shown in step 320 of fig. 3) one or more content items, each content item associated with a probability predicted by the machine learning model that is at least a threshold probability. In some embodiments, performance determination module 245 may do so by comparing the prediction probability associated with the content item to a threshold probability and then determining whether the prediction probability is at least the threshold probability based on the comparison. For example, assume that the prediction module 240 has applied a machine learning model to predict the probability that a plurality of content items are associated with a topic. In this example, the performance determination module 245 may compare the probability associated with each content item predicted by the machine learning model to a threshold probability. In this example, the performance determination module 245 identifies the content item if the probability is at least a threshold probability. In the above example, if the probability is less than the threshold probability, the performance determination module 245 does not identify the content item.

Performance determination module 245 also retrieves (e.g., as shown in step 325 of fig. 3) information describing user engagement with one or more content items. Examples of types of user engagement with a content item may include viewing the content item, sharing the content item, saving the content item, expressing a preference for the content item, commenting on the content item, reporting that the content item is not appropriate, interacting with the content item to unsubscribe from content created by an online system user that created the content item, and so forth. The performance determination module 245 may retrieve information describing user engagement with the content item from the action log 220 and/or the edge store 225. For example, the performance determination module 245 may retrieve information from the action log 220 and/or the side store 225 that describes the number of times a content item was viewed, the number of times a content item was shared, the number of times a content item was saved, the number of times one or more users of the online system 140 expressed a preference for a content item, the number of comments made to a content item, the time each user participated in a content item, and so forth. Alternatively, in the above example, performance determination module 245 may retrieve information describing each instance of the type of user engagement with the content item that occurred from action log 220 and/or edge store 225, which may then be tabulated to determine the number of times the type of user engagement with the content item occurred.

In some embodiments, the information describing user engagement with the content item retrieved by performance determination module 245 may also include various types of information associated with each online system user engaging in the content item. For example, the performance determination module 245 may retrieve information identifying each user participating in the content item, information describing the type of client device 110 associated with the user, and so forth. In embodiments where the performance determination module 245 retrieves information identifying each user participating in the content item, the performance determination module 245 may communicate the information to the audience determination module 250. In some embodiments, if performance determination module 245 determines that the value of the performance metric for the topic associated with the content item is at least a threshold value, performance determination module 245 may do so, as described further below. Alternatively, in some embodiments, if the performance determination module 245 determines that the value of the performance metric for the topic associated with the content item is anomalous (anomaly) (e.g., for the current time period), the performance determination module 245 may transmit information identifying each user participating in the content item to the audience determination module 250, as described further below.

The performance determination module 245 also determines (e.g., as shown in step 330 of fig. 3) a value of a performance metric for the topic based on information retrieved by the performance determination module 245 describing user engagement with one or more content items associated with the topic (e.g., content items associated with a probability predicted by the machine learning model that is at least a threshold probability). Examples of performance metrics include Click Through Rate (CTR), conversion rate, or any other suitable type of performance metric that describes user engagement with one or more content items. In embodiments where the performance determination module 245 determines the performance metric value for the topic based on information describing user engagement with a plurality of content items associated with the topic, the performance determination module 245 may determine the performance metric value for each content item and determine the performance metric value for the topic based on the performance metric values for all content items. For example, the performance determination module 245 may determine a value for the click-through rate of a topic as an average of the click-through rates of a plurality of content items associated with the topic.

Once the performance determination module 245 has determined the value of the performance metric for the topic, the performance determination module 245 may determine whether the value is at least a threshold value. The performance determination module 245 may do so by comparing the value of the performance metric for the topic to a threshold and then determining whether the value of the performance metric for the topic is at least the threshold based on the comparison. For example, assume that the performance determination module 245 has determined a value of a performance metric for a topic to be 3.5%, where the value of the performance metric corresponds to an average of landing page conversions for a plurality of content items associated with the topic. In this example, if the threshold corresponds to a value of 3.2%, the performance determination module 245 determines 335 that the value of the performance metric for the subject is at least the threshold, because 3.5% is equal to or greater than 3.2%.

In some embodiments, the performance determination module 245 may determine whether the value of the performance metric for the topic is abnormal. In such embodiments, the performance determination module 245 may make the determination for a particular time period (e.g., a current time period). For example, the performance determination module 245 may compare the performance metric value for the topic for the current time period to the performance metric value for the topic for the previous time period. In this example, the performance determination module 245 may determine that the value of the performance metric for the topic for the current time period is abnormal if the value of the performance metric for the topic for the current time period is greater than the value of the performance metric for the topic for the previous time period by at least a threshold amount. Alternatively, in the above example, if the value of the performance metric for the topic for the current time period is not greater than the value of the performance metric for the topic for the previous time period by at least a threshold amount, the performance determination module 245 may determine that the value of the performance metric for the topic for the current time period is not anomalous. The performance determination module 245 may use any anomaly detection technique (e.g., a density-based technique such as k-nearest neighbor, a type of support vector machine, etc.) to determine whether the value of the performance metric for the topic is an anomaly. The functionality of performance determination module 245 is further described below in conjunction with fig. 3.

Audience determination module 250 may determine a set of positioning criteria associated with a topic. To this end, the audience determination module 250 may receive information (e.g., from the performance determination module 245) identifying each online system user that participated in the content item associated with the topic. Audience determination module 250 may then retrieve various attributes associated with each user (e.g., from user profile store 205, action log 220, and/or side store 225) and determine a set of positioning criteria associated with the topic based on the retrieved information. For example, based on a username associated with each online system user participating in a content item associated with a topic, audience determination module 250 may retrieve information from user profile store 205 that describes demographic information associated with each user (e.g., age and geographic location associated with each user), information describing the hobbies and professions of each user, and so forth. In this example, the audience determination module 250 may also retrieve information from the action log 220 and/or the edge store 225 that describes each user's interactions (e.g., interactions with other users of the online system 140 or objects maintained in the online system 140). In the above example, based on the retrieved information, audience determination module 250 may determine a set of targeting criteria associated with the topic, where the targeting criteria are satisfied by users of online systems 140 that may have an affinity for the topic. The functionality of the audience determination module 250 is further described below in conjunction with fig. 3.

The user interface generator 255 generates recommendations to create content items. In some embodiments, the recommendation may correspond to a recommendation to create a content item describing the item. Further, in such embodiments, the items may be associated with a topic. In some embodiments, the recommendation may be included in an editor interface generated by user interface generator 255. In other embodiments, the recommendation may be included in a window (e.g., a pop-up window) or any other suitable user interface generated by user interface generator 255. The editor interface or other user interface including the recommendation may be transmitted to the user of the online system 140 or other entities having presence on the online system 140 via the client device 110 associated with the user/entity.

The recommendations generated by the user interface generator 255 may also or alternatively include one or more options to add various components to the content item. Examples of components that may be added to a content item include one or more tags, a title, one or more videos, one or more images, a set of targeting criteria that specify an audience for the content item, a location, one or more audio files, or any other suitable type of component. In embodiments where the recommendation includes one or more options for adding various components to the content item, the recommendation may also include pre-populated information corresponding to the components. For example, if the recommendation includes an option to add one or more tags to the content item, the recommendation may include tags associated with the object or subject, e.g., tags pre-populated with information describing the product corresponding to the object or information describing the product category corresponding to the subject. In the above example, if the recommendation further includes an option to add one or more images to the content item, the recommendation may be pre-populated with one or more images of the product included in the product catalog.

In embodiments where the recommendation includes pre-populated information corresponding to one or more components of the content item, the pre-populated information may be accepted or rejected. For example, assume that the online system 140 has transmitted a recommendation to an entity to create a content item and add a tag and an image to the content item, where the recommendation includes pre-populated information corresponding to the tag and the image. In this example, the entity may accept the pre-populated information by transmitting a request to create the content item to the presence system 140, where the request includes the pre-populated information. In this example, the online system 140 may then create a content item based on the request. Alternatively, in the above example, the entity may reject pre-populated information corresponding to one or more components (e.g., by deselecting or deleting tags and/or images). In this example, the entity may transmit a request to the presence system 140 to create the content item, where the request no longer includes the pre-populated information that was rejected by the entity. In this example, the online system 140 may then create a content item based on the request.

User interface generator 255 may generate a user interface that includes one or more values of one or more performance metrics for one or more topics. The user interface may correspond to a window (e.g., a pop-up window) that may be transmitted in conjunction with the recommendations generated by the user interface generator 255. In embodiments where the user interface generator 255 generates an editor interface that includes recommendations, values for performance metrics for the topics may also be included in the editor interface. The values of the performance metrics for a topic may be represented in various ways (e.g., by text, in a graphical form, etc.). For example, assume that the magnitude of the performance metric of the theme corresponds to the conversion rate of the tennis-related content, and assume that the additional value of the performance metric of the theme corresponds to the click-through rate of the tennis-related content. In this example, the user interface generator 255 may generate an editor interface or a separate window that includes a plurality of charts (graphs) indicating conversion rates and click-through rates for tennis-related content by users of the online system 140 who are presented with content items associated with the tennis theme. In some embodiments, the user interface/editor interface generated by the user interface generator 255 may include one or more interactive elements (e.g., drop down menus, buttons, etc.). In the above example, the interactive elements included in the editor interface/window may allow other entities presented with a chart or a user of the online system 140 to view a chart of values of performance metrics corresponding to different time periods, different geographic locations, and so forth.

In some embodiments, the user interface generator 255 may also generate content items based on requests received from a user of the online system 140 or any other entity having presence on the online system 140. The request may be received via a user interface (e.g., an editor interface) and may include one or more components specified in the request. The components of a content item may include a title, one or more images, one or more videos, one or more tags, a set of targeting criteria that specifies an audience for the content item, a location, one or more audio files, or any other suitable type of component. In some embodiments, the user interface generator 255 may generate the content item to include pre-populated information corresponding to one or more components of the content item that has been accepted by the online system user or other entity requesting the creation of the content item. In various embodiments, the user interface generator 255 may also or alternatively generate the content item to include information corresponding to one or more components of the content item that has been provided by the online system user or other entity requesting the creation of the content item. The functionality of user interface generator 255 is further described below in conjunction with fig. 3 and 4.

The content selection module 260 may identify one or more candidate content items that are eligible to be presented to the viewing user of the online system 140. The content selection module 260 retrieves candidate content items from the content store 210 or from another source that are eligible to be presented to the viewing user, and the content selection module 260 may rank the candidate content items and select one or more candidate content items for presentation to the viewing user. The candidate content items that qualify to be presented to the viewing user are content items that are associated with at least a threshold number of positioning criteria that are met by the characteristics of the viewing user, or are content items that are not associated with positioning criteria. In various embodiments, the content selection module 260 includes candidate content items that are eligible to be presented to the viewing user in one or more content selection processes that identify a group of content items for presentation to the viewing user. For example, the content selection module 260 determines a measure of relevance of various candidate content items to the viewing user based on characteristics of the online system 140 associated with the viewing user and based on the viewing user's affinity for different candidate content items. Based on the measure of relevance, the content selection module 260 selects content items for presentation to the viewing user. As a further example, the content selection module 260 selects the content item having the highest relevance metric, or at least a threshold relevance metric, for presentation to the viewing user. Optionally, the content selection module 260 ranks the candidate content items based on their associated relevance metrics and selects content items having a top position in the ranking or at least a threshold position in the ranking for presentation to the viewing user.

The content item selected for presentation to the viewing user may be associated with a bid amount. The content selection module 260 can use the bid amount associated with the candidate content item when selecting content for presentation to the viewing user. In various embodiments, the content selection module 260 determines the expected values associated with the various candidate content items based on the bid amounts of the various candidate content items, and selects the content item associated with the greatest expected value or associated with at least a threshold expected value for presentation to the viewing user. The expected value associated with the candidate content item represents an expected amount of compensation for presenting the candidate content item to the online system 140. For example, the expected value associated with the candidate content item is the product of the bid amount of the candidate content item and the likelihood that the viewing user interacts with the content from the candidate content item. The content selection module 260 can rank the candidate content items based on their associated bid amounts and select content items having at least a threshold position in the ranking for presentation to the viewing user. In some embodiments, the content selection module 260 ranks candidate content items not associated with a bid amount and candidate content items associated with a bid amount in a unified ranking based on the bid amount and relevance metric associated with the candidate content items. Based on the unified ranking, the content selection module 260 selects content for presentation to the viewing user. Selection of content items by unified ranking is further described in U.S. patent application No. 13/545,266, filed on 10/7/2012, which is incorporated herein by reference in its entirety.

For example, the content selection module 260 receives a request to present a content information stream (feed) to a viewing user of the online system 140. The information stream may include one or more advertisements as well as other content items, such as a dynamic (store) describing actions associated with other online system users connected to the viewing user. The content selection module 260 accesses one or more of the user profile store 205, the content store 210, the action log 220, and the side store 225 to retrieve information about the viewing user. For example, dynamic or other data associated with a user connected to the viewing user is retrieved. The content selection module 260 analyzes the retrieved dynamic or other content items to identify candidate content that may be relevant to the viewing user. For example, dynamics associated with users not associated with the viewing user or dynamics associated with users having less than a threshold affinity for the viewing user are discarded as candidate content. Based on various criteria, the content selection module 260 selects one or more content items identified as candidate content for presentation to the viewing user. The selected content item may be included in a content information stream presented to the viewing user. For example, the information stream of content includes at least a threshold number of content items that describe actions associated with users connected to the viewing user through the online system 140.

In various embodiments, the content selection module 260 presents content to the viewing user via a dynamic message (newsfeed) that includes a plurality of content items selected for presentation to the viewing user. One or more advertisements may also be included in the information stream. The content selection module 260 may also determine the order in which the selected content items are presented via the information stream. For example, the content selection module 260 orders the content items in the information stream based on the likelihood of the viewing user interacting with the various content items. The functionality of the content selection module 260 is further described below in conjunction with fig. 3.

The web server 265 links the online system 140 to one or more client devices 110 and to one or more third party systems 130 via the network 120. Web server 265 serves web pages and other content, e.g.XML, and the like. The web server 265 may receive and route messages (e.g., instant messages, queued messages (e.g., email), text messages, Short Message Service (SMS) messages, or messages sent using any other suitable messaging technique) between the presence system 140 and the client device 110. A user may send a request to web server 265 to upload information (e.g., images or videos) stored in content store 210. In addition, web server 265 may provide Application Programming Interface (API) functionality to send data directly to the native client device operating system, e.g.ANDROID^TM、Or blackberry os.

Suggesting entities in an online system to create items describing topics associated with at least a threshold of performance metrics The content of interest, and adding a tag describing the item to the content

FIG. 3 is a flow chart of a method for suggesting entities in an online system to create content that describes an item associated with a topic having at least a threshold value of a performance metric and to add a tag describing the item to the content. In other embodiments, the method may include steps different from and/or additional to those shown in FIG. 3. Additionally, the steps of the method may be performed in a different order than described in connection with fig. 3.

The online system 140 retrieves 305 (e.g., from the content store 210) a plurality of content items maintained in the online system 140 (e.g., using the prediction module 240). For example, the online system 140 accesses the content store 210 and retrieves 305 various content items created by one or more users of the online system 140. In some embodiments, the online system 140 may retrieve the content item (in step 305) based on a set of criteria (e.g., based on the creator of the content item, based on when the content item was created, based on a geographic location associated with the content item, etc.). For example, the online system 140 may retrieve (at step 305) only content items created by non-merchant users of the online system 140 within the past month. Alternatively, in the above example, the online system 140 may (at step 305) retrieve only content items that the merchant user of the online system 140 created within the past month.

The online system 140 then accesses 310 a trained machine learning model (e.g., a machine learning model trained by the machine learning module 235 or by the third party system 130) trained to predict a probability that the content item is associated with the topic (e.g., using the prediction module 240). Examples of topics that may be associated with a content item include people (e.g., private, public characters or groups), political questions or sports, organizations, specific items (e.g., specific products), categories of products or services, book types, dishes, meals, and so forth. As described above, the machine learning model may be a convolutional neural network, a deep learning model, or any other suitable machine learning model. In some embodiments, the online system 140 may access multiple machine learning models that collectively perform the functions of the machine learning model.

The machine learning model may be trained based on one or more topics associated with the training set of content items and content included in the training set of each content item. The content included in the content items that may be used to train the machine learning model may include various types of structured and/or unstructured data, such as text data, image data, video data, audio data, or any other suitable type of data that may be included in the content items. As described above, various techniques may be used to train a machine learning model. In embodiments where the content used to train the machine learning model includes textual data (e.g., text included in a title, comment, tag, metadata, or location associated with the content item), the machine learning model may be trained using one or more natural language processing techniques, such as a topic modeling algorithm and/or a Word embedding algorithm (e.g., Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (PLSA), Latent Dirichlet Allocation (LDA), Word2Vec, LDA2Vec, etc.). In embodiments where the content used to train the machine learning model includes audio data, the audio data may be pre-processed (e.g., by removing noise, transcribing speech into text, etc.). The machine learning model may also be trained using one or more algorithms that classify the audio data (e.g., into different types of noise, different types of music, etc.). Further, in embodiments where the audio data used to train the machine learning model includes speech transcribed into text, the text may be used to train the machine learning model using one or more natural language processing techniques, as described above. In embodiments where the content for training the machine learning model includes image data and/or video data, the machine learning model may be trained using one or more image processing techniques, as described below.

In embodiments where the content used to train the machine learning model includes image data and/or video data, the machine learning model may first detect one or more objects included in each content item retrieved 305 by the online system 140 and then predict a probability that each object is associated with a topic. The machine learning model may detect one or more objects included within an image in the content item (or included within an image in one or more frames of a video included in the content item) by applying one or more object detection methods to the image. The machine learning model may also identify the locations of objects detected in the image (e.g., by generating a bounding box around each object). In various embodiments, the machine learning model uses one or more object detection methods to identify a category or type corresponding to each object detected in the image. For example, using an object detection method, a machine learning model associates different categories or types (e.g., types of products or other items) with objects detected within an image based on the attributes of the objects. The machine learning model then makes one or more predictions, each prediction corresponding to a probability that an object detected in the image is associated with a subject (e.g., a category or type that the machine learning model has associated with the object), and outputs one or more probabilities. In various embodiments, the machine learning model may predict that an object detected in the image corresponds to a particular item (e.g., a product), where the item corresponds to a topic. In such embodiments, the machine learning model may compare the object to an image of the item (e.g., an image of a product included in a product catalog) and make one or more predictions based on the comparison, each prediction corresponding to a probability that the object detected in the image is associated with a topic.

In embodiments where the machine learning model predicts a probability that an object detected within an image included in a content item is associated with a topic, the machine learning model may be trained based on a set of training images that include images of different objects associated with different topics. In some embodiments, the set of training images may include publicly available information identifying different topics associated with different objects. The machine learning model may also be trained based on attributes (e.g., shapes, edges, curves, etc.) characterizing each training image set and information identifying topics associated with each training image set.

In embodiments where the machine learning model predicts a probability that an object detected within the image corresponds to a particular item, the machine learning model may be trained based on a comparison of the object detected within the image and the image of the item. In some embodiments, the machine learning model is trained to predict the probability that an object detected within an image matches an item based on previous matches of the object detected within the image to different items (e.g., images of different products included in one or more product categories). For example, when training the machine learning model, the online system 140 (or the third party system 130) applies a label to an object detected within the image based on the object's attributes (e.g., logo, trademark, logo, icon, pattern, texture, print, etc.) indicating that the object matches an item (e.g., product). The machine learning model is trained using any suitable training method or combination of training methods (e.g., using back propagation if the machine learning model is a neural network, curve fitting techniques if the machine learning model is a linear regression model, etc.) based on the labeled attributes of the objects extracted from the images.

The online system 140 then applies 315 a machine learning model (e.g., using the prediction module 240) to predict a probability that each content item retrieved 305 by the online system 140 is associated with a topic. To apply 315 the machine learning model to predict a probability that a content item retrieved 305 by the online system 140 is associated with a topic, the online system 140 provides input to the machine learning model that includes content of the content item. As described above, the content of a content item may include various types of structured and/or unstructured data (e.g., text data, image data, video data, audio data, etc.) included in one or more components of the content item (e.g., a title, one or more images, one or more videos, one or more tags, a set of targeting criteria that specifies an audience for the content item, a location, one or more audio files, etc.). Based on the content of the content item, the machine learning model predicts a probability that the content item is associated with the topic. The online system 140 then receives an output corresponding to the predicted probability from the machine learning model. In embodiments where the online system 140 retrieves multiple machine learning models that collectively perform the function of the machine learning model, the online system 140 may apply the machine learning models in a similar manner.

The online system 140 identifies 320 (e.g., using the performance determination module 245) one or more content items, each content item associated with a probability predicted by the machine learning model that is at least a threshold probability. In some embodiments, the online system 140 may do so by comparing the predicted probability associated with the content item to a threshold probability, and then determining whether the predicted probability is at least the threshold probability based on the comparison. For example, assume that the online system 140 has applied 315 a machine learning model to predict the probability that a content item retrieved 305 by the online system 140 is associated with a topic. In this example, the online system 140 can compare the probability associated with each content item predicted by the machine learning model to a threshold probability. In this example, the online system 140 identifies 320 the content item if the probability is at least a threshold probability. In the above example, if the probability is less than the threshold probability, the online system 140 does not identify 320 the content item.

In some embodiments, rather than accessing 310 and applying 315 a machine learning model, the online system 140 may determine one or more topics associated with each content item retrieved 305 by the online system 140 (e.g., using the topic extraction engine 230) and identify 320 the one or more content items associated with at least a threshold probability. For example, the online system 140 may determine one or more topics associated with each retrieved content item by identifying anchor terms included in the content item and then determining the meaning of each anchor term. In this example, each topic associated with a content item may then be stored in association with the content item (e.g., in the content store 210). Continuing the example, the online system 140 may identify each content item stored in association with a topic.

The online system 140 then retrieves 325 (e.g., using the performance determination module 245) information describing user engagement with the content item identified 320 by the online system 140 (e.g., from the action log 220 and/or the edge store 225). Examples of types of user engagement with a content item may include viewing the content item, sharing the content item, saving the content item, expressing a preference for the content item, commenting on the content item, reporting that the content item is not appropriate, interacting with the content item to unsubscribe from content created by an online system user creating the content item, and so forth. For example, the online system 140 may retrieve 325 information from the action log 220 and/or the side store 225 that describes the number of times each content item identified 320 by the online system 140 was viewed, the number of times each content item was shared, the number of times each content item was saved, the number of times one or more users of the online system 140 expressed a preference for each content item, the number of comments for each content item, the time each user participated in each content item, and so forth. Optionally, in the above example, the online system 140 may retrieve 325 from the action log 220 and/or the edge store 225 information describing each instance of the type of user engagement with each content item identified 320 by the online system 140 that occurred, which may then be tabulated to determine the number of times the type of user engagement with each content item occurred. In some embodiments, the information describing the user's participation in each content item retrieved 325 by the online system 140 that is identified 320 by the online system 140 may also include various types of information associated with each user participating in the content item. For example, the online system 140 may retrieve 325 information identifying each user participating in each content item identified 320 by the online system 140, information describing the type of client device 110 associated with the user, and so forth.

The online system 140 then determines 330 a value of a performance metric for the topic (e.g., using the performance determination module 245) based on the retrieved information describing the user's engagement with the content item. Examples of performance metrics include Click Through Rate (CTR), conversion rate, or any other suitable type of performance metric that describes user engagement with a content item. In embodiments where the presence system 140 determines 330 a value of a performance metric for the topic based on the information describing the user's engagement with the plurality of content items, the presence system 140 may determine the value of the performance metric for each content item and determine the value of the performance metric for the topic based on the values of the performance metrics for all content items. For example, the online system 140 may determine the value of the click-through rate for the topic as an average of the click-through rates for the plurality of content items identified 320 by the online system 140.

Once the presence system 140 has determined 330 the value of the performance metric for the topic, the presence system 140 may determine whether the value is at least a threshold value. The online system 140 may do so by comparing the value of the performance metric for the topic to a threshold and then determining whether the value of the performance metric for the topic is at least the threshold based on the comparison. For example, assume that the online system 140 has determined 330 that the value of the performance metric for the topic is 3.5%, where the value of the performance metric for the topic corresponds to an average of landing page conversions for a plurality of content items. In this example, if the threshold corresponds to a value of 3.2%, the online system 140 determines 335 that the value of the performance metric of the topic is at least the threshold, because 3.5% is equal to or greater than 3.2%.

In some embodiments, the online system 140 may determine (e.g., using the performance determination module 245) whether the value of the performance metric for the topic is abnormal. In such embodiments, the presence system 140 may make the determination for a particular time period (e.g., the current time period). For example, the online system 140 may compare the value of the performance metric for the topic for the current time period to the value of the performance metric for the topic for the previous time period. In this example, the online system 140 may determine that the value of the performance metric for the topic for the current time period is abnormal if the value of the performance metric for the topic for the current time period is greater than the value of the performance metric for the topic for the previous time period by at least a threshold amount. Optionally, in the above example, the online system 140 may determine that the value of the performance metric for the topic for the current time period is not abnormal if the value of the performance metric for the topic for the current time period is not greater than the value of the performance metric for the topic for the previous time period by at least a threshold amount. The online system 140 may use any anomaly detection technique (e.g., a density-based technique such as k-nearest neighbor, a class of support vector machines, etc.) to determine whether the value of the performance metric for the topic is an anomaly.

In some embodiments, the online system 140 may determine (e.g., using the audience determination module 250) a set of positioning criteria associated with the topic. The online system 140 may do so by retrieving various attributes associated with one or more users of the online system 140 that are engaged in each content item identified 320 by the online system 140. The online system 140 may then determine a set of location criteria associated with the topic based on the retrieved information. For example, the online system 140 may retrieve information from the user profile store 205 that describes demographic information associated with each user participating in each content item identified 320 by the online system 140 (e.g., age and geographic location associated with each user), information describing the hobbies and professions of each user, and so forth. In this example, the online system 140 may also retrieve information from the action log 220 and/or the edge store 225 that describes interactions of each user participating in each content item identified 320 by the online system 140 (e.g., interactions with other users of the online system 140 or objects maintained in the online system 140). In the above example, based on the retrieved information, the online system 140 may determine a set of location criteria associated with the topic, where the location criteria are satisfied by users of the online system 140 who may have an affinity for the topic. In some embodiments, if the presence system 140 determines 335 that the value of the performance metric for the topic is at least a threshold value, the presence system 140 may determine a set of positioning criteria associated with the topic. Alternatively, in some embodiments, if the online system 140 determines that the value of the performance metric is abnormal (e.g., for the current time period), the online system 140 may do so.

The presence system 140 receives 340 content describing an item from an entity having presence on the presence system 140, wherein the item is associated with a topic. In some embodiments, the content describing the item received 340 from the entity is a product catalog associated with the entity. In such embodiments, the product catalog may include various types of information associated with each product, such as a product identifier, one or more images of the product, a product name, a price provided by the entity for purchase of the product, a product description, and so forth. In some embodiments, the presence system 140 may store content received 340 from the entity (e.g., in a content store associated with the information identifying the entity). In various embodiments, the presence system 140 may receive information describing an item and/or topic from an entity. In various embodiments, the presence system 140 may determine that the content received 340 from the entity describes the item and/or that the item is associated with a topic. In such embodiments, the online system 140 may do so using one of the methods described above (e.g., by providing the content as input to a machine learning model, by determining topics associated with the content based on one or more anchor vocabularies included in the content, etc.).

If the presence system 140 determines 335 that the value of the performance metric for the topic is at least a threshold value (or determines that the value of the performance metric for the topic is abnormal), the presence system 140 determines 345 a transmission to the entity of the opportunity to create a recommendation for the content item describing the item. For example, assume that the online system 140 receives 340 a tennis product catalog from an entity that describes an item corresponding to a tennis racket, wherein the item is associated with a tennis theme. In this example, the online system 140 may determine 345, upon receiving a request to log into a user account associated with the entity, an opportunity to transmit a recommendation to the entity to create a content item describing a tennis racket. In some embodiments, the recommendation may also or alternatively include an option to add one or more components to the content item (e.g., a tag associated with the item). In embodiments where the recommendation includes only the option to add one or more components to the content item, the presence system 140 may determine an opportunity to deliver the recommendation to the entity upon receiving a request from the entity to create the content item. For example, assume that the presence system 140 receives a request from an entity to create a content item that describes an item associated with a topic. In this example, the online system 140 may determine, upon receiving the request, an opportunity to transmit a recommendation to the entity to add one or more images of the item to the content item.

Once the online system 140 has determined 345 an opportunity to communicate the recommendation to the entity, the online system 140 may generate the recommendation (e.g., using the user interface generator 255). In some embodiments, the recommendation may be included in an editor interface generated by the online system 140, while in other embodiments, the recommendation may be included in a window (e.g., a pop-up window) or other type of user interface generated by the online system 140. As described above, in some embodiments, a recommendation may include one or more options for adding one or more components to a content item (e.g., a tag associated with the item). Examples of components that may be added to a content item include one or more tags, a title, one or more videos, one or more images, a set of targeting criteria that specify an audience for the content item, a location, one or more audio files, or any other suitable type of component. In embodiments where the recommendation includes one or more options for adding one or more components to the content item, the recommendation may also include pre-populated information corresponding to the components.

Referring now to FIG. 4, assume that the subject corresponds to tennis and the item corresponds to a tennis racket, such that the online system 140 has generated a recommendation 405 for the entity to create a content item that describes the tennis racket. In this example, it is also assumed that the recommendation 405 is included in the editor interface 400, and that the recommendation 405 includes an option to add various components to the content item. In this example, the recommendation 405 may be pre-populated with an image 410 of a tennis racket (e.g., retrieved from a product catalog received 340 from the entity), a label 415 associated with the item (i.e., "# tennis racket (# tenisrake)"), location criteria 420A-420B that specify an audience for the content item (i.e., users between the ages of 25 and 34 having a hobby/interest including tennis), and a location 425 of the entity (i.e., "sporting goods store" in "state Z, city a").

In some embodiments, the online system 140 may also generate a user interface (e.g., a window) that includes values for performance metrics for the topic. In embodiments where the recommendation is included in an editor interface generated by the online system 140, the value of the performance metric for the topic may also be included in the editor interface. The value of the performance metric for a topic may be represented in various ways (e.g., by text, in a graphical form, etc.). As shown in the example of fig. 4, assume that the value of the performance metric for the topic corresponds to a conversion rate of tennis-related content. In this example, the editor interface 400 including the recommendation 405 may include an area 430 that displays the value of the performance metric for the subject using a plurality of graphs indicating conversion rates for tennis-related content by users of the online system 140 who were presented with content items associated with tennis subjects in various geographic locations (e.g., the united states, australia, etc.) for a period of time (e.g., the previous seven days) by the online system 140.

In some embodiments, the user interface or editor interface may include a plurality of values for one or more performance metrics for one or more topics. As shown in the example of fig. 4, if the added value of the performance metric for the theme corresponds to the click-through rate of tennis-related content, the area 430 of the editor interface 400 may also include a plurality of charts indicating the click-through rate of tennis-related content by users of the online system 140 who have been presented with content items associated with tennis themes in various geographic locations (e.g., the united states, australia, etc.) for a period of time (e.g., the previous seven days). In some embodiments, the user interface or editor interface may also include one or more interactive elements (e.g., drop down menus, buttons, etc.). In the example of fig. 4, the interactive elements included in the editor interface 400 may allow viewing of additional charts (e.g., charts of values of additional performance metrics corresponding to a topic, charts of values of performance metrics corresponding to different time periods, different geographic locations, different demographics, etc.).

Referring back to FIG. 3, the presence system 140 then transmits 350 the recommendation to the entity. For example, the online system 140 may transmit 350 the recommendation to the entity via the client device 110 associated with the entity. In embodiments where the online system 140 generates a user interface that includes values for performance metrics for a topic and that is separate from the user interface that includes recommendations, the values for performance metrics for the topic may be transmitted to the entity along with the recommendations.

The presence system 140 can then receive a request from an entity to create a content item, and the presence system 140 can then generate the content item based on the request (e.g., using the user interface generator 255). In some embodiments, the content item may include pre-populated information corresponding to one or more components of the content item that have been accepted by the entity. In embodiments where the recommendation includes pre-populated information corresponding to one or more components of the content item, the pre-populated information may be accepted or rejected. In the example of FIG. 4, the online system 140 may receive a request from an entity to create a content item upon receiving an interaction with the interactive element 435 of the editor interface 400. In this example, if, at the time the online system 140 receives the request, the editor interface 400 includes the pre-populated information for the component parts (i.e., the image 410, the tags 415, the targeting criteria 420A-420B that specify the audience for the content item, and the location 425), the pre-populated information is accepted and the content item may be created based on the request. Alternatively, in the above example, if some or all of the pre-populated information is removed from the editor interface when the online system 140 receives the request, the pre-populated information that has been removed is rejected, and the content item may be created based on the request. In various embodiments, the content item may also or alternatively include information corresponding to one or more components of the content item provided by the entity. In the above example, the content item may include a title added by the entity, as well as one or more additional images, tags, and positioning criteria.

The online system 140 may then send the content item for display to one or more viewing users of the online system 140. For example, in determining an opportunity to present content to a viewing user of the online system 140, the online system 140 may identify a set of candidate content items that are eligible to be presented to the viewing user (e.g., using the content selection module 260) based on the positioning criteria associated with the content items. If a content item is included in the set of candidate content items, the online system 140 may rank (e.g., using the content selection module 260) the content item in the set of candidate content items based on the targeting criteria associated with the content item and the targeting criteria associated with each additional content item included in the set of candidate content items. The online system 140 then selects one or more content items from the set of candidate content items for presentation to the viewing user based on the rankings (e.g., using the content selection module 260), and sends the selected content items for presentation to the viewing user.

Summary of the invention

The foregoing description of the embodiments has been presented for purposes of illustration; it is not intended to be exhaustive or to limit the patent to the precise form disclosed. One skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of the present description describe embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Moreover, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code, which may be executed by a computer processor, for performing any or all of the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes, and/or it may comprise a general purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Moreover, any computing system referred to in the specification may include a single processor, or may be an architecture that employs a multi-processor design to increase computing power.

Embodiments may also relate to products produced by the computing processes described herein. Such products may include information produced by a computing process, where the information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other combination of data described herein.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the patent rights be limited not by this detailed description, but rather by any claims issued on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the patent, which is set forth in the following claims.