阅读说明：本技术 一种视频播放方法和装置 (Video playing method and device ) 是由 付万豪 刘殿超 张观良 赵颖 杨光伟 李壮 于 2019-03-29 设计创作，主要内容包括：本发明公开了一种视频播放方法和装置。本发明的视频播放方法,通过获取对目标区域进行拍摄得到的视频,目标区域包括一个或多个目标物。根据视频的拍摄信息对视频进行目标物识别,建立识别出的目标物与目标区域的全景图中对应目标物的映射关系。基于映射关系,在视频的播放过程中结合全景图进行目标物的可视化展示。这样可以更加清楚地了解视频中的目标物在现实中的位置,使目标物更加直观。根据拍摄信息对目标物进行识别,可以使目标物的识别更加准确。同时,通过目标物控制视频播放,更容易进行目标物的播放展示,而不需要通过调整进度条对目标物进行寻找,使视频的播放体验更加好,增加视频播放控制的多元化。(The invention discloses a video playing method and device. According to the video playing method, the video obtained by shooting the target area is obtained, and the target area comprises one or more target objects. And carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and the corresponding target object in the panoramic image of the target area. And based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video. Therefore, the position of the target object in the video in reality can be more clearly known, and the target object is more visual. The target object is identified according to the shooting information, so that the identification of the target object can be more accurate. Meanwhile, the video playing is controlled through the target object, the playing and displaying of the target object are easier to perform, the target object does not need to be found through adjusting the progress bar, the playing experience of the video is better, and the diversification of the video playing control is increased.)

1. A video playback method, comprising:

acquiring a video obtained by shooting a target area, wherein the target area comprises one or more target objects;

carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and a corresponding target object in a panoramic image of a target area;

and based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video.

2. The method of claim 1, wherein the identifying the video according to the shooting information of the video, and the establishing the mapping relationship between the identified object and the corresponding object in the panorama of the target area comprises:

selecting a corresponding type of recognition algorithm according to the type corresponding to each target object in the target area;

when a first target object of a specified type is identified according to an identification algorithm, determining the geographical position information of the target object according to the shooting information;

determining a second target object matched with the first target object in the panoramic image according to the geographic position information of the first target object and the geographic position information of each target object in the panoramic image;

and establishing a mapping relation between the first target object and the second target object.

3. The method of claim 1, wherein the performing, based on the mapping relationship, a visual display of the object in combination with the panorama during the playing of the video comprises:

and displaying the panoramic image in the video playing process, and marking a corresponding target object in the panoramic image according to the mapping relation when the target object appears in the video.

4. The method according to claim 1, wherein the mapping relation stores a time point at which the identified object appears in the video, shot information corresponding to the time point, and geographic position information of the corresponding object in the panorama.

5. The method of claim 4, further comprising:

constructing a shooting track according to the shooting information, and displaying the shooting track in the panoramic image;

the visual display of the target object in combination with the panoramic image in the playing process of the video based on the mapping relationship comprises:

responding to a selection request of a track point on the shooting track, determining a corresponding time point according to the mapping relation, and skipping the video to the determined time point for playing.

6. The method of claim 4, wherein the performing, based on the mapping relationship, the visual display of the object in combination with the panorama during the playing of the video comprises:

responding to a selection request of a specified target object in the panoramic image, determining a corresponding time point according to the mapping relation, and skipping the video to the determined time point for playing.

7. The method of claim 1, wherein the video is captured by an onboard camera;

the shooting information comprises posture information and/or geographical position information when the camera shoots.

8. A video playback apparatus, comprising:

the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a video obtained by shooting a target area, and the target area comprises one or more target objects;

the establishing unit is used for identifying a target object of the video according to the shooting information of the video and establishing a mapping relation between the identified target object and the corresponding target object in the panorama of the target area;

and the display unit is used for carrying out visual display on the target object by combining the panoramic image in the playing process of the video based on the mapping relation.

9. An electronic device, wherein the electronic device comprises: a processor; and a memory arranged to store computer-executable instructions that, when executed, cause the processor to perform the method of any one of claims 1-7.

10. A computer readable storage medium, wherein the computer readable storage medium stores one or more programs which, when executed by a processor, implement the method of any of claims 1-7.

Technical Field

The invention relates to the technical field of video playing, in particular to a video playing method and device.

Background

Disclosure of Invention

In view of the above, the present invention has been made to provide a video playing method and apparatus that overcome the above problems or at least partially solve the above problems.

According to an aspect of the present invention, there is provided a video playing method, including:

acquiring a video obtained by shooting a target area, wherein the target area comprises one or more target objects;

carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and a corresponding target object in a panoramic image of a target area;

and based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video.

Optionally, the performing, according to the shooting information of the video, object identification on the video, and establishing a mapping relationship between the identified object and a corresponding object in a panorama of a target area includes:

selecting a corresponding type of recognition algorithm according to the type corresponding to each target object in the target area;

when a first target object of a specified type is identified according to an identification algorithm, determining the geographical position information of the target object according to the shooting information;

determining a second target object matched with the first target object in the panoramic image according to the geographic position information of the first target object and the geographic position information of each target object in the panoramic image;

and establishing a mapping relation between the first target object and the second target object.

Optionally, the performing, based on the mapping relationship, visual display of the target object in combination with the panoramic image in the playing process of the video includes:

and displaying the panoramic image in the video playing process, and marking a corresponding target object in the panoramic image according to the mapping relation when the target object appears in the video.

Optionally, the mapping relationship stores a time point at which the identified target object appears in the video, shooting information corresponding to the time point, and geographic position information of the corresponding target object in the panorama.

Optionally, the method further comprises:

constructing a shooting track according to the shooting information, and displaying the shooting track in the panoramic image;

the visual display of the target object in combination with the panoramic image in the playing process of the video based on the mapping relationship comprises:

responding to a selection request of a track point on the shooting track, determining a corresponding time point according to the mapping relation, and skipping the video to the determined time point for playing.

Optionally, the performing, based on the mapping relationship, visual display of the target object in combination with the panoramic image in the playing process of the video includes:

responding to a selection request of a specified target object in the panoramic image, determining a corresponding time point according to the mapping relation, and skipping the video to the determined time point for playing.

Optionally, the video is captured by an onboard camera;

the shooting information comprises posture information and/or geographical position information when the camera shoots.

According to another aspect of the present invention, there is provided a video playback apparatus, including:

the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring a video obtained by shooting a target area, and the target area comprises one or more target objects;

the establishing unit is used for identifying a target object of the video according to the shooting information of the video and establishing a mapping relation between the identified target object and the corresponding target object in the panorama of the target area;

and the display unit is used for carrying out visual display on the target object by combining the panoramic image in the playing process of the video based on the mapping relation.

In accordance with still another aspect of the present invention, there is provided an electronic apparatus including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform a method as any one of the above.

According to a further aspect of the invention, there is provided a computer readable storage medium, wherein the computer readable storage medium stores one or more programs which, when executed by a processor, implement a method as any one of the above.

As can be seen from the above, according to the technical solution of the present invention, a video obtained by shooting a target area is obtained, and the target area includes one or more target objects. And carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and the corresponding target object in the panoramic image of the target area. And based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video. The target object in the video and the target object in the target area are mapped, so that the position of the target object in the video in reality can be more clearly known, and the target object is more visual. The target object is identified according to the shooting information, so that the identification of the target object can be more accurate. Meanwhile, the video playing is controlled through the target object, the playing and displaying of the target object are easier to perform, the target object does not need to be found through adjusting the progress bar, the playing experience of the video is better, and the diversification of the video playing control is increased.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

fig. 1 is a flow chart illustrating a video playing method according to an embodiment of the present invention;

FIG. 2A illustrates a video playback screen and a display view of a panorama according to one embodiment of the present invention;

FIG. 2B illustrates a presentation of a video playback screen, an infrared screen, and a panorama in accordance with one embodiment of the present invention;

fig. 3 is a schematic structural diagram of a video playback apparatus according to an embodiment of the present invention;

FIG. 4 shows a schematic structural diagram of an electronic device according to one embodiment of the invention;

fig. 5 shows a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Fig. 1 is a flow chart illustrating a video playing method according to an embodiment of the present invention. As shown in fig. 1, the method includes:

step S110, a video obtained by shooting a target area is acquired, and the target area includes one or more target objects.

The target area is an area to be monitored, the target object in the target area is a monitoring object, and videos of the target area are shot so that the situation of each target object can be known from the videos. For example, in a photovoltaic power plant, an area where a photovoltaic panel is arranged is photographed to know the condition of the photovoltaic panel, such as the presence or absence of a breakage or the like, from a photographed video. Wherein each photovoltaic panel is the target. The video can be obtained by various shooting modes, such as shooting through a camera mounted on a unmanned aerial vehicle.

And step S120, carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and a corresponding target object in a panoramic image of the target area.

In order to solve the problem that the playing position of a video can only be changed by adjusting a progress bar during video playing, the invention provides that the playing position of the video is adjusted by a target object. Therefore, the target object in the video needs to be identified, and a mapping relation between the playing position of the video and the target object is established, so that the playing position of the video is determined directly through the target object. For example, clicking the target object causes the video to jump to the time point when the corresponding target object appears for playing.

That is, the target object in the entire video is identified, and the information of the target object and the playing position information of the target object appearing in the video are recorded. In order to make the target object more intuitive, the target object in the video and the target object in the target area are mapped, and specifically, the target object is mapped into the panorama of the target area, so that the position of the target object in the video in reality can be more clearly understood, and the target object is more definite. The panorama of the target area can be obtained through a satellite map, or obtained through high-altitude shooting of the target area by an aircraft, or spliced after low-altitude shooting of the target area by an unmanned aerial vehicle, or a virtual drawing drawn manually.

The shooting information of the video specifically refers to information when the video is shot, such as a shooting position of the video, a shooting time of the video, and the like. And carrying out object identification on the video according to the shooting information of the video, namely determining an object in the video according to the shooting position or the shooting time of the video. When the target object is identified, each frame of the video can be identified, and the target object and frame information in each frame are recorded until all frames of the video are identified.

Taking the video of the photovoltaic panel as an example, since the structures of the photovoltaic panels are the same, when no identification mark is set on each photovoltaic panel, it is difficult for a user to determine the position of the photovoltaic panel in the current video picture in reality. Therefore, the photovoltaic panel in each frame of the video is identified, a mapping relation is established between the photovoltaic panel in the video and the corresponding photovoltaic panel in the panoramic image of the power station, for example, the photovoltaic panel in the video is identified to be the photovoltaic panel No. 1, and the photovoltaic panel in the video is mapped to the position of the photovoltaic panel No. 1 in the panoramic image, so that the position of the photovoltaic panel in the video in reality can be determined, and the information displayed by the video is clearer and more intuitive.

And step S130, based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video.

And displaying the panoramic image of the target area while playing the video, and enabling the target objects in the current video image to correspond to the target objects in the panoramic image one by one. The panoramic view serves as an auxiliary display, wherein the positions of the various objects in the target area are displayed. And the target object in the video and the corresponding target object in the panoramic image are correspondingly displayed, so that the user can know the target object more intuitively. In addition, the video can be controlled to jump to the position of the corresponding target object appearing in the video for playing according to the mapping relation of the target object in the panoramic image through the target object selected in the panoramic image. Therefore, the target object can be played and displayed more easily, the target object does not need to be searched by adjusting the progress bar, and the playing experience of the video is better.

On the other hand, the information of the target object can also be directly displayed, for example, the photovoltaic panel 1 to 10 is shot by a video, and the information of the photovoltaic panel 1 to 10 is displayed while the video is played, so that the visualization of the target object is realized. Meanwhile, if one or more photovoltaic panels are selected, the video is controlled to jump to the position where the corresponding photovoltaic panel appears to be played based on the corresponding mapping relation, and the diversification of video playing control is increased.

According to the technical scheme, the video obtained by shooting the target area is obtained, and the target area comprises one or more target objects. And carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and the corresponding target object in the panoramic image of the target area. And based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video. The target object in the video and the target object in the target area are mapped, so that the position of the target object in the video in reality can be more clearly known, and the target object is more visual. The target object is identified according to the shooting information, so that the identification of the target object can be more accurate. Meanwhile, the video playing is controlled through the target object, the playing and displaying of the target object are easier to perform, the target object does not need to be found through adjusting the progress bar, the playing experience of the video is better, and the diversification of the video playing control is increased.

In an embodiment of the present invention, as in the method shown in fig. 1, in the step S120, performing object identification on the video according to the shooting information of the video, and establishing a mapping relationship between the identified object and a corresponding object in the panorama of the target area includes: selecting a corresponding type of recognition algorithm according to the type corresponding to each target object in the target area; when a first target object of a specified type is identified according to an identification algorithm, determining the geographical position information of the target object according to the shooting information; determining a second target object matched with the first target object in the panoramic image according to the geographical position information of the first target object and the geographical position information of each target object in the panoramic image; and establishing a mapping relation between the first target object and the second target object.

Different target areas have different target objects, and when the target objects are identified, corresponding identification algorithms are needed for identification. For example, the structure of the photovoltaic panel is different from that of the wind driven generator, and a corresponding recognition algorithm needs to be selected to improve the recognition accuracy.

For the purpose of discrimination, the object in the video is referred to as a first object, and the object in the panorama is referred to as a second object. When the first target object is identified, determining the mapping relation between the first target object and a second target object in the panoramic image according to the geographic position information of the first target object. The geographical location information includes longitude, latitude, and the like. The geographic position information of each target object is unique, so that the accuracy of the established mapping relation can be ensured.

The geographic position information of each target object in the panoramic image can be obtained through a positioning system, and the geographic position information of the target objects in the video is determined through shooting information. For example, videos are obtained by shooting of the unmanned aerial vehicle, a GPS is arranged in the unmanned aerial vehicle, and in the flying shooting process of the unmanned aerial vehicle, the GPS can record geographic position information of the unmanned aerial vehicle. According to the geographical position information of the unmanned aerial vehicle, the geographical position information of the target object in the video can be determined.

Taking the above photovoltaic panel as an example, the geographic location information of the number 1 to 10 photovoltaic panels in the target area has been determined by the positioning system. The camera carried on the unmanned aerial vehicle shoots the video vertically downwards, so that the geographic position information of the unmanned aerial vehicle is the geographic position information of the corresponding target object. Specifically, a picture of one photovoltaic panel in a first frame of the video is identified, and the shooting time of the first frame is 10: 00. At 10:00, the geographic position information of the unmanned aerial vehicle is the same as that of the No. 1 photovoltaic panel, so that the photovoltaic panel in the first frame of video is determined to be the No. 1 photovoltaic panel, and the first frame of the video and the No. 1 photovoltaic panel are mapped. And by analogy, establishing a mapping relation of No. 2 to No. 10 photovoltaic panels.

On the other hand, when the camera mounted on the drone does not shoot vertically downward but has a certain shooting angle, the geographic position information of the GPS positioning of the drone may not correspond to the geographic position information of the target object in the video picture at this time. Thus, it is necessary to determine the geographical location information of the object in the video frame through more shooting information. Specifically, the unmanned aerial vehicle is generally provided with an attitude sensor for acquiring attitude data of the unmanned aerial vehicle, and angular velocity, acceleration and the like of the unmanned aerial vehicle in the current state are acquired. For example, attitude data such as a pitch angle, a yaw angle, and a roll angle of the drone is measured by an inertial measurement sensor or a gyroscope, a distance between the drone and a target is measured by a radar on the drone, and the like. And determining the direction of the target object in the video picture according to the attitude data of the unmanned aerial vehicle, and determining the geographical position information of the target object by combining the distance measured by the radar and the geographical position information positioned by the GPS.

Considering that a plurality of objects may exist in a video picture at the same time, in order to establish a more accurate mapping relationship, the plurality of objects are distinguished. For example, there are two photovoltaic panels in the first frame of the video. The shooting information also includes parameters of a camera for shooting the video, such as parameters of the focal length, depth of field and the like of the camera. Determining the range of the picture shot by the camera according to the parameters of the camera, and combining the range with the geographical position information to determine two photovoltaic panels in the first frame picture. For example, the size of the range shot by the camera is determined to be 5m × 5m, and at the position determined by the geographic position information, the photovoltaic panels within the 5m × 5m range include the photovoltaic panel No. 1 and the photovoltaic panel No. 2, so that the two photovoltaic panels in the first frame picture are respectively mapped with the photovoltaic panel No. 1 and the photovoltaic panel No. 2.

Certainly, there is also a situation that there is no corresponding photovoltaic panel under the corresponding geographic location information, but the photovoltaic panel is identified in the video picture, and at this time, the specific information of the photovoltaic panel in the video picture can also be determined by adopting the above-mentioned method for determining the photovoltaic panel according to the shooting range of the camera, which is not described herein again.

In an embodiment of the present invention, in the method shown in fig. 1, based on the mapping relationship in step S130, performing visual display on the target object in combination with the panorama during the playing of the video includes: and displaying the panoramic image in the video playing process, and marking a corresponding target object in the panoramic image according to the mapping relation when the target object appears in the video.

The panorama is used as an auxiliary display mode, so that a user can be helped to know the target object more intuitively. Specifically, as shown in fig. 2A, the video playing frame 210 and the panoramic view 220 are displayed simultaneously, and when an object appears in the video playing frame 210, the corresponding object is marked in the panoramic view 220 according to the corresponding mapping relationship, for example, the corresponding object is framed by a frame, so that the user can watch the object more clearly. If a plurality of objects may exist in the video picture at the same time, all the objects are marked out in the panoramic image at the same time according to the corresponding mapping relation. In addition, the video shot by the specific camera is combined, and the video pictures shot by the specific camera are synchronously displayed during displaying. For example, fig. 2B shows a video playing picture 210 and a panorama 220 captured by a normal camera, and an infrared picture 230 captured by an infrared camera, wherein the infrared picture 230 shows infrared information of a current target object. Of course, the object can also be photographed by an ultraviolet camera or a multispectral camera.

In one embodiment of the present invention, as shown in fig. 1, the mapping relationship stores the time point of the identified object appearing in the video, the shooting information corresponding to the time point, and the geographic location information of the corresponding object in the panorama.

The mapping relation stores corresponding target object information and other information. Specifically, the mapping relationship stores a time point at which the target object appears in the video, shooting information corresponding to the time point, and geographic position information corresponding to the target object in the panorama. According to the information stored in the mapping relation, the video playing can be controlled through the target object.

Still taking the photovoltaic panel as an example, the mapping relationship of the photovoltaic panel No. 1 includes the time point of the photovoltaic panel No. 1 appearing in the video, for example, the photovoltaic panel No. 1 appears in the video of the 1 st frame and the 50 th frame, and the time point of the video corresponding to each frame is 00:01, 10:06, and the like. Meanwhile, shooting information when the 1 st frame is shot, shooting information when the 50 th frame is shot, and longitude and latitude of the geographic position information of the No. 1 photovoltaic panel are also obtained.

According to the mapping relation, when the No. 1 photovoltaic panel in the panoramic image is selected, the video jumps to 00:01 to be played; when the No. 1 photovoltaic panel is selected again, the video jumps to 10:06 to be played.

In one embodiment of the present invention, as in the method shown in fig. 1, the method further comprises: and constructing a shooting track according to the shooting information, and displaying the shooting track in the panoramic image. Based on the mapping relationship in step S130, performing visual display of the target object in combination with the panorama in the playing process of the video includes: responding to a selection request of a track point on the shooting track, determining a corresponding time point according to the mapping relation, and skipping the video to the determined time point for playing.

As mentioned in the above embodiments, the video is captured by a camera mounted on the drone, and the captured information includes information about the geographical location of the drone during flight. Therefore, the flight track of the unmanned aerial vehicle can be constructed according to the geographical position information so as to obtain the shooting track of the video. The shooting track is combined with the panoramic image, so that the shooting process is more intuitive, and the user can know the approximate time of each target object in the video from the combined image. Meanwhile, the user can control the video to jump to the corresponding time point for playing by selecting one track point on the shooting track.

Each track point corresponds to geographical position information, the mapping relation with the same geographical position information is found out, and the video is skipped to the time point for playing according to the time point of the stored video. Therefore, the video playing is controlled through the shooting track, and the progress bar does not need to be adjusted for many times to find out the target object to be watched.

Considering that the accuracy of the geographical position information of the track points is high, the mapping relation with the identical geographical position information may not be matched. At this time, a mapping relation with geographical position information closest to the track point is matched to solve the problem that the mapping relation cannot be matched.

In an embodiment of the present invention, in the method, based on the mapping relationship in step S130, performing visual display on the target object in combination with the panorama during the playing of the video includes: responding to a selection request of a specified target object in the panoramic image, determining a corresponding time point according to the mapping relation, and skipping the video to the determined time point for playing.

In addition to the above embodiments, the video playing may be controlled by selecting the track point, and the video playing may also be controlled by responding to a selection request for the target object. For example, when the No. 1 photovoltaic panel is selected, the time point of the No. 1 photovoltaic panel appearing in the video is determined according to the mapping relation of the No. 1 photovoltaic panel, and the video is controlled to jump to the corresponding time point for playing, for example, the video jumps to 00:01 for playing; when the No. 1 photovoltaic panel is selected again, the video jumps to 10:06 to be played.

In one embodiment of the invention, as in the method shown in FIG. 1, the video is taken by an onboard camera; the shooting information includes pose information and/or geographical position information at the time of shooting by the camera.

Airborne camera, the camera that is to say embarks on unmanned aerial vehicle. Attitude information is the attitude data of unmanned aerial vehicle, such as data such as angular velocity, acceleration, pitch angle, yaw angle of unmanned aerial vehicle, detects through the last corresponding sensor of unmanned aerial vehicle and obtains, and geographical position information is obtained by the GPS location on the unmanned aerial vehicle. Unmanned aerial vehicle is at the flight in-process, and the machine carries the camera and shoots the video to the target object in the target area, and sensor detection obtains unmanned aerial vehicle's attitude information, and GPS fixes a position unmanned aerial vehicle, obtains geographical position information. Each piece of information comprises corresponding time information, and the video can be in one-to-one correspondence with the attitude information and the geographic position information through the time information so as to obtain an accurate mapping relation. For example, according to the time when the first frame of video is shot, the attitude information and the geographical position information at that time are acquired, so that the mapping relation between the target object in the first frame of video and the corresponding target object in the panorama is determined.

Fig. 3 is a schematic structural diagram of a video playback apparatus according to an embodiment of the present invention. As shown in fig. 3, the apparatus 300 includes:

the acquiring unit 310 is configured to acquire a video obtained by shooting a target area, where the target area includes one or more target objects.

The target area is an area to be monitored, the target object in the target area is a monitoring object, and videos of the target area are shot so that the situation of each target object can be known from the videos. For example, in a photovoltaic power plant, an area where a photovoltaic panel is arranged is photographed to know the condition of the photovoltaic panel, such as the presence or absence of a breakage or the like, from a photographed video. Wherein each photovoltaic panel is the target. The video can be obtained by various shooting modes, such as shooting through a camera mounted on a unmanned aerial vehicle.

The establishing unit 320 is configured to perform object identification on the video according to the shooting information of the video, and establish a mapping relationship between the identified object and a corresponding object in the panorama of the target area.

In order to solve the problem that the playing position of a video can only be changed by adjusting a progress bar during video playing, the invention provides that the playing position of the video is adjusted by a target object. Therefore, the target object in the video needs to be identified, and a mapping relation between the playing position of the video and the target object is established, so that the playing position of the video is determined directly through the target object. For example, clicking the target object causes the video to jump to the time point when the corresponding target object appears for playing.

That is, the target object in the entire video is identified, and the information of the target object and the playing position information of the target object appearing in the video are recorded. In order to make the target object more intuitive, the target object in the video and the target object in the target area are mapped, and specifically, the target object is mapped into the panorama of the target area, so that the position of the target object in the video in reality can be more clearly understood, and the target object is more definite. The panorama of the target area can be obtained through a satellite map, or obtained through high-altitude shooting of the target area by an aircraft, or spliced after low-altitude shooting of the target area by an unmanned aerial vehicle, or a virtual drawing drawn manually.

The shooting information of the video specifically refers to information when the video is shot, such as a shooting position of the video, a shooting time of the video, and the like. And carrying out object identification on the video according to the shooting information of the video, namely determining an object in the video according to the shooting position or the shooting time of the video. When the target object is identified, each frame of the video can be identified, and the target object and frame information in each frame are recorded until all frames of the video are identified.

Taking the video of the photovoltaic panel as an example, since the structures of the photovoltaic panels are the same, when no identification mark is set on each photovoltaic panel, it is difficult for a user to determine the position of the photovoltaic panel in the current video picture in reality. Therefore, the photovoltaic panel in each frame of the video is identified, a mapping relation is established between the photovoltaic panel in the video and the corresponding photovoltaic panel in the panoramic image of the power station, for example, the photovoltaic panel in the video is identified to be the photovoltaic panel No. 1, and the photovoltaic panel in the video is mapped to the position of the photovoltaic panel No. 1 in the panoramic image, so that the position of the photovoltaic panel in the video in reality can be determined, and the information displayed by the video is clearer and more intuitive.

The display unit 330 is configured to perform visual display of the target object in combination with the panorama in the playing process of the video based on the mapping relationship.

And displaying the panoramic image of the target area while playing the video, and enabling the target objects in the current video image to correspond to the target objects in the panoramic image one by one. The panoramic view serves as an auxiliary display, wherein the positions of the various objects in the target area are displayed. And the target object in the video and the corresponding target object in the panoramic image are correspondingly displayed, so that the user can know the target object more intuitively. In addition, the video can be controlled to jump to the position of the corresponding target object appearing in the video for playing according to the mapping relation of the target object in the panoramic image through the target object selected in the panoramic image. Therefore, the target object can be played and displayed more easily without searching the target object by adjusting the progress bar, so that the playing experience of the video is better.

On the other hand, the information of the target object can also be directly displayed, for example, the photovoltaic panel 1 to 10 is shot by a video, and the information of the photovoltaic panel 1 to 10 is displayed while the video is played, so that the visualization of the target object is realized. Meanwhile, if one or more photovoltaic panels are selected, the video is controlled to jump to the position where the corresponding photovoltaic panel appears to be played based on the corresponding mapping relation, and the diversification of video playing control is increased.

According to the technical scheme, the video obtained by shooting the target area is obtained, and the target area comprises one or more target objects. And carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and the corresponding target object in the panoramic image of the target area. And based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video. The target object in the video and the target object in the target area are mapped, so that the position of the target object in the video in reality can be more clearly known, and the target object is more visual. The target object is identified according to the shooting information, so that the identification of the target object can be more accurate. Meanwhile, the video playing is controlled through the target object, the playing and displaying of the target object are easier to perform, the target object does not need to be found through adjusting the progress bar, the playing experience of the video is better, and the diversification of the video playing control is increased.

In an embodiment of the present invention, as in the apparatus 300 shown in fig. 3, the establishing unit 320 is configured to select a corresponding type of recognition algorithm according to a type corresponding to each target object in the target area; when a first target object of a specified type is identified according to an identification algorithm, determining the geographical position information of the target object according to the shooting information; determining a second target object matched with the first target object in the panoramic image according to the geographical position information of the first target object and the geographical position information of each target object in the panoramic image; and establishing a mapping relation between the first target object and the second target object.

Different target areas have different target objects, and when the target objects are identified, corresponding identification algorithms are needed for identification. For example, the structure of the photovoltaic panel is different from that of the wind driven generator, and a corresponding recognition algorithm needs to be selected to improve the recognition accuracy.

For the purpose of discrimination, the object in the video is referred to as a first object, and the object in the panorama is referred to as a second object. When the first target object is identified, determining the mapping relation between the first target object and a second target object in the panoramic image according to the geographic position information of the first target object. The geographical location information includes longitude, latitude, and the like. The geographic position information of each target object is unique, so that the accuracy of the established mapping relation can be ensured.

The geographic position information of each target object in the panoramic image can be obtained through a positioning system, and the geographic position information of the target objects in the video is determined through shooting information. For example, videos are obtained by shooting of the unmanned aerial vehicle, a GPS is arranged in the unmanned aerial vehicle, and in the flying shooting process of the unmanned aerial vehicle, the GPS can record geographic position information of the unmanned aerial vehicle. According to the geographical position information of the unmanned aerial vehicle, the geographical position information of the target object in the video can be determined.

Taking the above photovoltaic panel as an example, the geographic location information of the number 1 to 10 photovoltaic panels in the target area has been determined by the positioning system. The camera carried on the unmanned aerial vehicle shoots the video vertically downwards, so that the geographic position information of the unmanned aerial vehicle is the geographic position information of the corresponding target object. Specifically, a picture of one photovoltaic panel in a first frame of the video is identified, and the shooting time of the first frame is 10: 00. At 10:00, the geographic position information of the unmanned aerial vehicle is the same as that of the No. 1 photovoltaic panel, so that the photovoltaic panel in the first frame of video is determined to be the No. 1 photovoltaic panel, and the first frame of the video and the No. 1 photovoltaic panel are mapped. And by analogy, establishing a mapping relation of No. 2 to No. 10 photovoltaic panels.

On the other hand, when the camera mounted on the drone does not shoot vertically downward but has a certain shooting angle, the geographic position information of the GPS positioning of the drone may not correspond to the geographic position information of the target object in the video picture at this time. Thus, it is necessary to determine the geographical location information of the object in the video frame through more shooting information. Specifically, the unmanned aerial vehicle is generally provided with an attitude sensor for acquiring attitude data of the unmanned aerial vehicle, and angular velocity, acceleration and the like of the unmanned aerial vehicle in the current state are acquired. For example, attitude data such as a pitch angle, a yaw angle, and a roll angle of the drone is measured by an inertial measurement sensor or a gyroscope, a distance between the drone and a target is measured by a radar on the drone, and the like. And determining the direction of the target object in the video picture according to the attitude data of the unmanned aerial vehicle, and determining the geographical position information of the target object by combining the distance measured by the radar and the geographical position information positioned by the GPS.

Considering that a plurality of objects may exist in a video picture at the same time, in order to establish a more accurate mapping relationship, the plurality of objects are distinguished. For example, there are two photovoltaic panels in the first frame of the video. The shooting information also includes parameters of a camera for shooting the video, such as parameters of the focal length, depth of field and the like of the camera. Determining the range of the picture shot by the camera according to the parameters of the camera, and combining the range with the geographical position information to determine two photovoltaic panels in the first frame picture. For example, the size of the range shot by the camera is determined to be 5m × 5m, and at the position determined by the geographic position information, the photovoltaic panels within the 5m × 5m range include the photovoltaic panel No. 1 and the photovoltaic panel No. 2, so that the two photovoltaic panels in the first frame picture are respectively mapped with the photovoltaic panel No. 1 and the photovoltaic panel No. 2.

Certainly, there is also a situation that there is no corresponding photovoltaic panel under the corresponding geographic location information, but the photovoltaic panel is identified in the video picture, and at this time, the specific information of the photovoltaic panel in the video picture can also be determined by adopting the above-mentioned method for determining the photovoltaic panel according to the shooting range of the camera, which is not described herein again.

In an embodiment of the present invention, as in the apparatus 300 shown in fig. 3, the displaying unit 330 is configured to display a panorama during playing of a video, and mark a corresponding target object in the panorama according to a mapping relationship when the target object appears in the video.

The panorama is used as an auxiliary display mode, so that a user can be helped to know the target object more intuitively. Specifically, as shown in fig. 2A, the video playing frame 210 and the panoramic view 220 are displayed simultaneously, and when an object appears in the video playing frame 210, the corresponding object is marked in the panoramic view 220 according to the corresponding mapping relationship, for example, the corresponding object is framed by a frame, so that the user can watch the object more clearly. If a plurality of objects may exist in the video picture at the same time, all the objects are marked out in the panoramic image at the same time according to the corresponding mapping relation. In addition, the video shot by the specific camera is combined, and the video pictures shot by the specific camera are synchronously displayed during displaying. For example, fig. 2B shows a video playing picture 210 and a panorama 220 captured by a normal camera, and an infrared picture 230 captured by an infrared camera, wherein the infrared picture 230 shows infrared information of a current target object. Of course, the object can also be photographed by an ultraviolet camera or a multispectral camera.

In one embodiment of the present invention, as shown in the apparatus 300 shown in fig. 3, the mapping relationship stores the time point of the identified object appearing in the video, the shooting information corresponding to the time point, and the geographic location information of the corresponding object in the panorama.

The mapping relation stores corresponding target object information and other information. Specifically, the mapping relationship stores a time point at which the target object appears in the video, shooting information corresponding to the time point, and geographic position information corresponding to the target object in the panorama. According to the information stored in the mapping relation, the video playing can be controlled through the target object.

Still taking the photovoltaic panel as an example, the mapping relationship of the photovoltaic panel No. 1 includes the time point of the photovoltaic panel No. 1 appearing in the video, for example, the photovoltaic panel No. 1 appears in the video of the 1 st frame and the 50 th frame, and the time point of the video corresponding to each frame is 00:01, 10:06, and the like. Meanwhile, shooting information when the 1 st frame is shot, shooting information when the 50 th frame is shot, and longitude and latitude of the geographic position information of the No. 1 photovoltaic panel are also obtained.

According to the mapping relation, when the No. 1 photovoltaic panel in the panoramic image is selected, the video jumps to 00:01 to be played; when the No. 1 photovoltaic panel is selected again, the video jumps to 10:06 to be played.

In one embodiment of the present invention, as in the apparatus 300 shown in fig. 3, the apparatus further comprises: and the track unit is used for constructing a shooting track according to the shooting information and displaying the shooting track in the panoramic image. The display unit 330 is further configured to determine a corresponding time point according to the mapping relationship in response to a selection request for a track point on the shooting track, and skip the video to the determined time point for playing.

As mentioned in the above embodiment, the video is captured by a camera mounted on the drone, and the captured information includes information on the geographical location of the drone during flight. Therefore, the flight track of the unmanned aerial vehicle can be constructed according to the geographical position information, so that the shooting track of the video is obtained. The shooting track is combined with the panoramic image, so that the shooting process is more intuitive, and the user can know the approximate time of each target object in the video from the combined image. Meanwhile, the user can control the video to jump to the corresponding time point for playing by selecting one track point on the shooting track.

Each track point corresponds to geographical position information, the mapping relation with the same geographical position information is found out, and the video is skipped to the time point for playing according to the time point of the stored video. Therefore, the video playing is controlled through the shooting track, and the progress bar does not need to be adjusted for many times to find out the target object to be watched.

Considering that the accuracy of the geographical position information of the track points is high, the mapping relation with the identical geographical position information may not be matched. At this time, a mapping relation with geographical position information closest to the track point is matched to solve the problem that the mapping relation cannot be matched.

In an embodiment of the present invention, in the apparatus 300, the displaying unit 330 is further configured to determine, in response to a selection request for a specified target object in the panoramic image, a corresponding time point according to the mapping relationship, and jump the video to the determined time point for playing.

In addition to the above embodiments, the video playing may be controlled by selecting the track point, and the video playing may also be controlled by responding to a selection request for the target object. For example, when the No. 1 photovoltaic panel is selected, the time point of the No. 1 photovoltaic panel appearing in the video is determined according to the mapping relation of the No. 1 photovoltaic panel, and the video is controlled to jump to the corresponding time point for playing, for example, the video jumps to 00:01 for playing; when the No. 1 photovoltaic panel is selected again, the video jumps to 10:06 to be played.

In one embodiment of the present invention, as in the device 300 shown in FIG. 3, the video is captured by an onboard camera; the shooting information includes pose information and/or geographical position information at the time of shooting by the camera.

Airborne camera, the camera that is to say embarks on unmanned aerial vehicle. Attitude information is the attitude data of unmanned aerial vehicle, such as data such as angular velocity, acceleration, pitch angle, yaw angle of unmanned aerial vehicle, detects through the last corresponding sensor of unmanned aerial vehicle and obtains, and geographical position information is obtained by the GPS location on the unmanned aerial vehicle. Unmanned aerial vehicle is at the flight in-process, and the machine carries the camera and shoots the video to the target object in the target area, and sensor detection obtains unmanned aerial vehicle's attitude information, and GPS fixes a position unmanned aerial vehicle, obtains geographical position information. Each piece of information comprises corresponding time information, and the video can be in one-to-one correspondence with the attitude information and the geographic position information through the time information so as to obtain an accurate mapping relation. For example, according to the time when the first frame of video is shot, the attitude information and the geographical position information at that time are acquired, so that the mapping relation between the target object in the first frame of video and the corresponding target object in the panorama is determined.

In summary, according to the technical solution of the present invention, a video obtained by shooting a target area is obtained, where the target area includes one or more target objects. And carrying out target object identification on the video according to the shooting information of the video, and establishing a mapping relation between the identified target object and the corresponding target object in the panoramic image of the target area. And based on the mapping relation, carrying out visual display on the target object by combining the panoramic image in the playing process of the video. The target object in the video and the target object in the target area are mapped, so that the position of the target object in the video in reality can be more clearly known, and the target object is more visual. The target object is identified according to the shooting information, so that the identification of the target object can be more accurate. Meanwhile, the video playing is controlled through the target object, the playing and displaying of the target object are easier to perform, the target object does not need to be found through adjusting the progress bar, the playing experience of the video is better, and the diversification of the video playing control is increased.

It should be noted that:

the algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose devices may be used with the teachings herein. The required structure for constructing such a device will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a video playback device in accordance with embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

For example, fig. 4 shows a schematic structural diagram of an electronic device according to an embodiment of the invention. The electronic device comprises a processor 410 and a memory 420 arranged to store computer executable instructions (computer readable program code). The memory 420 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. The memory 420 has a storage space 430 storing computer readable program code 431 for performing any of the method steps described above. For example, the storage space 430 for storing the computer readable program code may include respective computer readable program codes 431 for respectively implementing various steps in the above method. The computer readable program code 431 can be read from or written to one or more computer program products. These computer program products comprise a program code carrier such as a hard disk, a Compact Disc (CD), a memory card or a floppy disk. Such a computer program product is typically a computer readable storage medium such as described in fig. 5. Fig. 5 shows a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention. The computer readable storage medium 500 has stored thereon a computer readable program code 431 for performing the steps of the method according to the invention, which is readable by the processor 410 of the electronic device 400, the computer readable program code 431, when executed by the electronic device 400, causing the electronic device 400 to perform the steps of the method described above, in particular the computer readable program code 431 stored thereon, is capable of performing the method shown in any of the embodiments described above. The computer readable program code 431 may be compressed in a suitable form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.