阅读说明：本技术 图像记录装置、图像记录方法以及记录介质 (Image recording apparatus, image recording method, and recording medium ) 是由 林宏之 于 2019-05-21 设计创作，主要内容包括：本发明提供能够抑制编码量的增加的图像记录装置、图像记录方法以及程序。形成动态图像的帧的序列按每规定周期包含帧内预测帧、帧间预测帧、及非参照帧,在从指示静态图像的拍摄的指示时间点到下一个非参照帧的待机期间为规定的时间以下时,对该下一个非参照帧所对应的静态图像进行编码,在所述待机期间超过所述规定的时间时,将所述当前帧的图像作为非参照帧进行编码,并且对所述当前帧所对应的静态图像进行编码。(The invention provides an image recording apparatus, an image recording method, and a program capable of suppressing an increase in the code amount. A sequence of frames forming a moving image includes an intra-frame prediction frame, an inter-frame prediction frame, and a non-reference frame every predetermined period, and when a standby period from an instruction time point for instructing the shooting of a still image to a next non-reference frame is a predetermined time or less, a still image corresponding to the next non-reference frame is encoded, and when the standby period exceeds the predetermined time, an image of the current frame is encoded as a non-reference frame and a still image corresponding to the current frame is encoded.)

1. an image recording apparatus, characterized by comprising:

A moving image encoding unit that encodes an image of each frame forming a moving image;

A still image encoding unit that encodes a still image; and

a control part for controlling the operation of the display device,

the sequence of frames forming the dynamic image includes:

An intra-frame prediction frame for the moving image encoding unit to perform intra-frame prediction and encode; and

an inter-frame prediction frame which is encoded by performing inter-frame prediction with reference to an image of another frame as a reference image,

The sequence includes, in the intra-frame prediction frames, non-reference frames that are not referred to in inter-frame prediction of a past inter-frame prediction frame, at every predetermined cycle,

In the control section, the control section controls the operation of the motor,

When the standby period from the instruction time point of the instruction of the shooting of the still image to the next non-reference frame is less than or equal to the specified time, the still image coding unit codes the still image corresponding to the next non-reference frame,

When the standby period exceeds the predetermined time, the moving image encoding unit encodes the image of the current frame at the instructed time point as a non-reference frame and the still image encoding unit encodes the still image corresponding to the current frame, or,

When the standby period is less than the predetermined time, causing the still image encoding unit to encode the still image corresponding to the next non-reference frame,

When the standby period is equal to or longer than the predetermined time, the moving image encoding unit encodes the image of the current frame as a non-reference frame, and the still image encoding unit encodes a still image corresponding to the current frame.

2. The image recording device according to claim 1,

In the control section, the control section controls the operation of the motor,

The predetermined time period is shortened as the motion amount of the moving image is larger.

3. The image recording device according to claim 2,

In the control section, the control section controls the operation of the motor,

Obtaining a motion vector representing a motion of an image in a block of at least a portion of the current frame from other frames,

Determining the amount of motion based on the motion vector.

4. The image recording device according to claim 2,

In the control section, the control section controls the operation of the motor,

recognizing a predetermined subject from the image of each frame,

Determining a representative value of a motion vector for each block in a display area of the object as the motion amount.

5. The image recording device according to claim 1,

An acceleration detecting unit for detecting an acceleration is provided,

In the control section, the control section controls the operation of the motor,

The amount of movement of the present apparatus is determined based on the acceleration,

The predetermined time period is shortened as the amount of exercise of the present apparatus is larger.

6. The image recording device according to any one of claims 1 to 5,

in the control section, the control section controls the operation of the motor,

an evaluation value representing the acceptability of the subject to be captured as the moving image is calculated,

it is determined whether the still image needs to be captured based on the evaluation value.

7. An image recording method for an image recording apparatus, comprising:

A moving image encoding process of encoding an image of each frame forming a moving image;

a still image encoding process of encoding a still image; and

The control process is carried out by controlling the process,

The sequence of frames forming the dynamic image includes:

an intra-frame prediction frame that performs intra-frame prediction and performs encoding in the moving image encoding process; and an inter-frame prediction frame for encoding by performing inter-frame prediction while referring to an image of another frame as a reference image,

The sequence includes, in the intra-frame prediction frames, non-reference frames that are not referred to in inter-frame prediction of a past inter-frame prediction frame, at every predetermined cycle,

In the course of the control described above,

when the standby period from the instruction time point of the instruction of the shooting of the still image to the next non-reference frame is below a predetermined time, the still image corresponding to the next non-reference frame is coded in the still image coding process,

When the standby period exceeds the predetermined time, the image of the current frame at the instructed time point is encoded as a non-reference frame in the moving image encoding process, and a still image corresponding to the current frame is encoded in the still image encoding process, or,

When the standby period is shorter than the predetermined time, the still picture corresponding to the next non-reference frame is encoded in the still picture encoding process,

When the standby period is equal to or longer than the predetermined time, the image of the current frame is encoded as a non-reference frame in the moving image encoding process, and a still image corresponding to the current frame is encoded in the still image encoding process.

8. A recording medium having a program recorded thereon, the program causing a computer of an image recording apparatus to execute:

A moving image encoding step of encoding an image of each frame forming a moving image;

A still image encoding step of encoding a still image; and

the control step is that the control step is carried out,

the sequence of frames forming the dynamic image includes:

An intra-frame prediction frame which performs intra-frame prediction and performs encoding in the moving image encoding step; and an inter-frame prediction frame for encoding by performing inter-frame prediction while referring to an image of another frame as a reference image,

The sequence includes, in the intra-frame prediction frames, non-reference frames that are not referred to in inter-frame prediction of a past inter-frame prediction frame, at every predetermined cycle,

In the step of controlling,

When the standby period from the instruction time point for instructing the shooting of the still image to the next non-reference frame is less than or equal to a predetermined time, the still image coding step codes the still image corresponding to the next non-reference frame,

When the standby period exceeds the predetermined time, the image of the current frame at the instructed time point is encoded as a non-reference frame in the moving image encoding step, and a still image corresponding to the current frame is encoded in the still image encoding step, or,

When the standby period is shorter than the predetermined time, the still picture corresponding to the next non-reference frame is encoded in the still picture encoding step,

When the standby period is equal to or longer than the predetermined time, the image of the current frame is encoded in the moving image encoding step as a non-reference frame, and the still image corresponding to the current frame is encoded in the still image encoding step.

Technical Field

The present invention relates to an image recording apparatus, an image recording method, and a recording medium for simultaneously recording a moving image and a still image.

Background

In an image recording apparatus that records a captured image as digital data, a recording method has been proposed in which a video and a still image in a specific case are recorded simultaneously.

for example, in the recording method described in patent document 1, a predetermined operation is received during recording of a video, an I picture is generated when the operation is performed, and the generated I picture is recorded as a still image.

The I picture is an image of a frame to be encoded by performing intra prediction in the present frame without referring to images of other frames among images of a plurality of frames constituting a video. When playing a video from a frame corresponding to a recorded still image, the frame corresponding to the still image is an I picture, and therefore the amount of processing required for the playing processing is reduced compared to performing playing from another frame.

Disclosure of Invention

Technical problem to be solved by the invention

On the other hand, at the time of playing of video, it is necessary that the capacity (encoding amount) of encoded data obtained by encoding does not exceed the capacity of the buffer of the decoder. The capacity of encoded data of an I picture is generally larger than that of encoded data of an image of a frame to be encoded (for example, a B picture or a P picture) by performing inter-frame prediction with reference to another frame. In the recording method described in patent document 1, an I picture is generated at a timing instructed by an operation, and the encoded data is recorded, so that the amount of encoded data increases. As a result, the amount of coding of the portion exceeds the capacity of the decoder buffer, and there is a possibility that the video cannot be normally played.

The present invention has been made in view of the above-described problems, and an object thereof is to provide an image recording apparatus, an image recording method, and a recording medium capable of suppressing an increase in the code amount.

Means for solving the problems

The present invention has been made to solve the above-described problems, and one aspect of the present invention is an image recording apparatus including: a moving image encoding unit that encodes an image of each frame forming a moving image; a still image encoding unit that encodes a still image; and a control unit for forming a sequence of frames of the moving image, the control unit including: an intra-frame prediction frame for the moving image encoding unit to perform intra-frame prediction and encode; and an inter-frame prediction frame which is encoded by referring to images of other frames as reference images and performing inter-frame prediction, wherein the sequence includes, in each predetermined cycle, a non-reference frame which is not referred to in inter-frame prediction of a past inter-frame prediction frame, and wherein the control unit causes the still image encoding unit to encode a still image corresponding to a next non-reference frame when a standby period from an instruction time point for instructing the capturing of a still image to the next non-reference frame is a predetermined time or less, and causes the moving image encoding unit to encode a still image corresponding to the current frame using an image of the current frame at the instruction time point as a non-reference frame when the standby period exceeds the predetermined time, or causes the still image encoding unit to encode a still image corresponding to the current frame, or when the standby period is less than the predetermined time, and a still image encoding unit configured to encode a still image corresponding to the next non-reference frame, wherein when the standby period is equal to or longer than the predetermined time, the still image encoding unit is configured to encode the image of the current frame as a non-reference frame and the still image encoding unit is configured to encode the still image corresponding to the current frame.

Effects of the invention

According to the present invention, an increase in the amount of code can be suppressed.

Drawings

Fig. 1 is a block diagram showing an example of a functional configuration of an image recording apparatus according to the present embodiment.

Fig. 2 is an explanatory diagram illustrating an operation example of the image recording apparatus according to the present embodiment.

Fig. 3 is a flowchart showing an example of the image recording process according to the present embodiment.

fig. 4 is a diagram showing an example of screen display according to the present embodiment.

Fig. 5 is a block diagram showing another example of the functional configuration of the image recording apparatus according to the present embodiment.

Detailed Description

Hereinafter, an image recording apparatus, an image recording method, and a recording medium according to embodiments of the present invention will be described with reference to the drawings.

Fig. 1 is a block diagram showing an example of a functional configuration of an image recording apparatus 10 according to the present embodiment.

The image recording apparatus 10 captures an image, encodes image data representing the captured image, and records encoded data obtained by the encoding. The image recording apparatus 10 is capable of capturing and recording moving images and still images. Further, the image recording apparatus 10 may capture a still image during the capture of a moving image. The image recording apparatus 10 may be a dedicated imaging apparatus (camera), or may be mounted as an electronic device having another function, for example, a multi-function mobile phone, a tablet terminal apparatus, or the like.

when a period from an instruction time point for instructing the shooting of a still image to a next non-reference frame (described later) is within a predetermined period, the image recording apparatus 10 records the still image of the next non-reference frame. When the period from the instructed time point to the next non-reference frame exceeds a predetermined period, the image recording apparatus 10 records the image of the frame instructed time point as a still image and performs moving image coding on the image of the frame as a non-reference frame.

The image recording apparatus 10 includes: an operation unit 101, a first image capturing unit 102, a second image capturing unit 103, a control unit 104, a moving image encoding unit 105, a still image encoding unit 106, a recording unit 107, an image decoding unit 111, a display unit 112, and a communication unit 113.

The operation unit 101 receives an operation by a user, generates an operation signal corresponding to the received operation, and outputs the generated operation signal to the control unit 104. The operation unit 101 includes a member for receiving an operation, such as a button, a dial, and a lever. The operation unit 101 may be a touch sensor mounted on the display unit 112 in a stacked manner. The functions of the image recording apparatus 10 are controlled by operating the operation unit 101. For example, shooting or shooting stop of a moving image, shooting of a still image, display or display stop of a recorded moving image, display of a still image, and the like are instructed.

the first image capturing unit 102 captures an image indicating an object existing in a visual field range, and outputs first image data indicating the captured image to the moving image encoding unit 105. In the following description, an image captured by the first imaging unit 102 is referred to as a first image. The first image is a moving image formed of images photographed for each predetermined frame. The first imaging unit 102 includes an optical system, an imaging element, and a signal processing unit (not shown). The optical system includes an objective lens. The objective lens converges the image light incident on the objective lens and causes the image light to be incident on the surface of the image pickup element. The imaging element is configured such that a plurality of cells are arranged on the surface thereof at a constant period in the horizontal direction and the vertical direction. Here, the horizontal direction and the vertical direction refer to a direction on one side and a direction on the other side of an imaging region formed in a plane perpendicular to an optical axis of the optical system. The other direction is a direction orthogonal to the one direction. The cell corresponds to each pixel and has a light receiving element. The light receiving element generates an electric signal having a voltage corresponding to the intensity of the received light as a signal value. The light receiving element is, for example, a CCD (Charge Coupled Device), a CMOS (Complementary-Metal-Oxide-Semiconductor), or the like.

The signal processing unit generates first image data representing a first image at a predetermined frame period (for example, 1/30 seconds). The first image is represented by a signal value per pixel. The optical system of the first imaging unit 102 has a wide angle of view (also referred to as a field angle, for example, 135 °). The optical system of the first imaging unit 102 may have a wide depth of field, and may realize a wide focus (also referred to as a deep focus) without performing focus control (also referred to as auto focus control, af (automatic focusing) control, focusing, or the like).

The second imaging unit 103 images an image and outputs second image data representing the imaged image to the still image encoding unit 106. In the following description, an image captured by the second imaging unit 103 is referred to as a second image. The second image is a still image indicating a point in time of shooting. The second imaging unit 103 includes an optical system, an imaging element, and a signal processing unit, as in the first imaging unit 102. However, the angle of view of the optical system of the second imaging unit 103 may be narrower than the angle of view of the optical system of the first imaging unit 102. The depth of field of the optical system of the second imaging unit 103 may be shallower than the depth of field of the optical system of the first imaging unit 102.

the second image pickup unit 103 may include a focus control unit (not shown) that can vary the distance between the objective lens and the image pickup device. The signal processing unit of the second imaging unit 103 generates second image data representing a second image captured by the imaging element and outputs the second image data to the focus control unit. The second image is represented by signal values per pixel. The focus control unit calculates a focus evaluation value indicating a degree of focus (hereinafter, referred to as a degree of focus) based on the signal value per pixel indicated by the second image data, and performs focus control by changing the position of the objective lens so that the degree of focus indicated by the focus evaluation value is higher. The focus evaluation value is, for example, a ratio of the intensity of a low-frequency component having a shortage of a predetermined spatial frequency and the intensity of a high-frequency component having a predetermined spatial frequency or higher, and the intensity of the high-frequency component to the intensity of the low-frequency component, calculated in the second image. The focus control unit specifies the position of the objective lens having a very high degree of focus. The signal processing unit outputs a focus completion signal indicating completion of the focus control to the signal processing unit of the second imaging unit 103. When an imaging control signal instructing to image a still image is input from the imaging control unit 121, the signal processing unit of the second imaging unit 103 outputs the received second image data representing the second image to the still image encoding unit 106 for a predetermined shutter speed (also referred to as an exposure time).

In addition, the imaging element of the second imaging unit 103 may capture the second image regardless of whether the imaging instruction is given or not. Further, the focus control unit may perform focus control on the second image captured.

The control unit 104 controls the operations of the respective units of the image recording apparatus 10. The control unit 104 includes an imaging control unit 121 and a display control unit 122. The control unit 104 may be a computer including one or more processors (e.g., a cpu (central Processing unit)). The control unit 104 may realize the functions of the imaging control unit 121 and the display control unit 122 by executing a process instructed by a command described in a program stored in advance in a storage medium. A program for realizing the functions of these components may be recorded on a computer-readable recording medium, and the processor may cause a computer system to read and execute the program recorded on the recording medium.

the imaging control unit 121 controls the function of imaging based on the operation signal input from the operation unit 101. The imaging control unit 121 controls, for example, the imaging of the moving image or the stop of the imaging by the first imaging unit 102 and the moving image encoding unit 105 and the imaging of the still image by the second imaging unit 103 and the still image encoding unit 106 based on the operation signal.

The display control unit 122 controls a display function based on an operation signal input from the operation unit 101. The display control unit 122 controls, for example, the display or stop of the moving image or the display of the still image by the image decoding unit 111 and the display unit 112 based on the operation signal. Examples of the processing performed by the photographing control section 121 and the display control section 122 will be described below.

The moving image encoding unit 105 generates first encoded data by performing encoding processing on the first image data input from the first imaging unit 102. In the encoding process, an encoding method corresponding to a moving image decoding method defined by an arbitrary standard such as MPEG (moving Picture Expert group) -1, MPEG-4AVC (advanced Video Coding), MPEG-H HEVC (High Efficiency Video Coding) or the like may be adopted. By encoding in these encoding methods, the amount of information of the first image can be compressed. The moving image encoding unit 105 sequentially records the generated first encoded data in the recording unit 107.

The still image encoding unit 106 generates second encoded data by performing encoding processing on the second image data input from the second imaging unit 103. Each primary encoding process is second image data representing a still image of 1 frame. In the encoding process, a still Image encoding method defined by an arbitrary standard, such as jpeg (joint Photographic Experts group), png (portable Network graphics), gif (graphics Interchange format), and heif (high Efficiency Image File format), may be used. By encoding in these still image encoding methods, the amount of information of the second image can be compressed. The still image encoding unit 106 records the generated second encoded data in the recording unit 107. The still image encoding unit 106 may associate the second encoded data with a frame of the first image captured simultaneously with the second image when the second encoded data is recorded in the recording unit 107.

The recording unit 107 records first encoded data representing a first image and second encoded data representing a second image. The recording unit 107 may be any nonvolatile memory such as a flash memory that does not erase recorded data even when power supply is stopped. The recording unit 107 may be fixed to a housing (not shown) of the image recording apparatus 10, or may be constituted by a recording medium that is detachable from the housing, and an interface to which the recording medium is attached. As the detachable recording medium, for example, an SD (Secure Digital) memory card may be used.

The image decoding unit 111 decodes the first encoded data or the second encoded data under the control of the control unit 104. For example, when an operation signal instructing display of an image or an image to be displayed is input from the operation unit 101, the control unit 104 generates a decoding control signal indicating decoding of first encoded data or second encoded data representing the image. The image decoding unit 111 reads the first encoded data or the second encoded data indicated by the decoding control signal from the recording unit 107 via the control unit 104.

When the decoding control signal instructs the first encoded data, the image decoding unit 111 decodes the first encoded data to generate first image data, and outputs the generated first image data to the display unit 112. The decoding method used for decoding the first encoded data may be any decoding method corresponding to the video encoding method used for encoding the first encoded data.

when the decoding control signal instructs the second encoded data, the image decoding unit 111 decodes the read second encoded data to generate second image data, and outputs the generated second image data to the display unit 112.

The decoding method used for decoding the second encoded data may be any decoding method corresponding to the still image encoding method used for encoding the second encoded data.

the display unit 112 displays various visual information based on the control of the control unit 104. The Display portion 112 may be any one of a Liquid Crystal Display (LCD), an Organic Electro-luminescence Display (OLED), and the like.

When the first image data is input from the image decoding unit 111, the display unit 112 displays a first image (moving image) indicated by the first image data. When the second image data is input from the image decoding unit 111, the display unit 112 displays a second image (still image) indicated by the second image data.

Further, the first imaging unit 102 may output the first image data to the display unit 112 when a display control signal instructing display of the first image is input from the control unit 104 in response to input of an operation signal indicating display of the first image from the operation unit 101. The display unit 112 displays the first image (moving image) during the shooting of the first image.

The communication unit 113 performs communication with another device by wireless or wired communication using a predetermined communication method. The communication unit 113 is, for example, a communication interface. For example, when an operation signal instructing transmission of an image, an image to be transmitted, and a device to which the image is to be transmitted is input from the operation unit 101, the control unit 104 generates a communication control signal indicating transmission of the first encoded data or the second encoded data showing the image to the device to which the image is to be transmitted. When the communication control signal is input from the control unit 104, the communication unit 113 reads one or both of the first encoded data and the second encoded data indicated by the communication control signal from the recording unit 107, and transmits the read data to a device (not shown) of a transmission destination indicated by the communication control signal. The destination device may include: a decoding unit that decodes data acquired from the image recording apparatus 10; and a display unit that displays an image based on the image data obtained by the decoding.

(example of operation)

Next, an operation example of the image recording apparatus 10 according to the user operation will be described.

the example shown in fig. 2 mainly shows the operation of the image recording apparatus 10 related to shooting. The horizontal axis of fig. 2 represents time t. The more rightward relative to fig. 2 indicates the more elapsed time. Fig. 2 shows the states of the user operation, the control unit 104 (imaging control unit 121), the moving image encoding unit 105, and the still image encoding unit 106 in the order from top to bottom.

at time t0, when the start of power supply (power on) is instructed by the user operation to the operation unit 101, the power supply unit (not shown) of the image recording apparatus 10 starts supplying power to each unit including the control unit 104. After that, each unit is started.

when an operation signal instructing the start of shooting of a moving image is input from the operation unit 101 in response to a user operation at time t1, the shooting control unit 121 of the control unit 104 outputs a shooting control signal indicating the start of shooting of a moving image to the first shooting unit 102. The first imaging unit 102 starts imaging of a first image, which is a moving image, and starts outputting first image data representing the imaged first image to the moving image encoding unit 105. The moving image encoding unit 105 starts encoding the first image data input from the first imaging unit 102. The moving image encoding unit 105 sequentially records the first encoded data obtained by encoding in the recording unit 107.

A moving image is generally represented by a sequence of a plurality of images. Each image is called a frame. The sequence of frames includes an intra-frame prediction frame and an inter-frame prediction frame, depending on the type of frame. An intra-frame prediction frame is a frame to be subjected to intra-frame prediction in encoding or decoding. Intra prediction is a process of determining a signal value of a pixel to be processed from a signal value of a processed pixel in a frame to be processed without referring to an image of another frame, and is also called intra (intra) prediction. Intra-predicted frames are also referred to as I-frames. The image of the I frame corresponds to the I picture.

An inter-prediction frame is a frame to be subjected to inter-prediction in encoding or decoding. Inter prediction is a process of specifying a signal value per pixel of an image of a frame to be processed with reference to a decoded image of another processed frame, and is also called inter prediction. Among inter prediction, there are forward prediction and bi-directional prediction. The forward prediction is a prediction method in which a frame to be referred to is a past frame compared with a frame to be processed. Bidirectional prediction is a prediction method in which a frame to be referred to is both a past frame and a future frame compared to a frame to be processed. The frame to be forward predicted and the image of the frame are referred to as a P frame and a P picture, respectively. The frames to be bi-directionally predicted and the images of the frames are referred to as B frames and B pictures, respectively. In the prediction process for the B frame in the encoding or decoding process, a plurality of previous frames are also referred to as reference images.

The moving image encoding unit 105 performs encoding processing in units of a frame group including a predetermined number of frames equal to or greater than 2, based on the above-described encoding method. Each group is composed of a sequence of individual frame classes. This group is for example called GOP (group of Pictures) in MPEG-4AVC or SOP (Structure of Pictures) in MPEG-H HEVC. Each group includes at least one non-reference frame. The non-reference frame is a type of I frame and is not referred to in inter prediction of inter prediction frames in a past group. The non-reference frame is called an idr (instant Decoder refresh) frame. A general example is that the non-reference frame is the first frame of the group. The other frames in the group become any of I frames, P frames, or B frames. The non-reference frame is referred to in inter prediction of other frame prediction frames of a group to which the frame belongs. "I" of fig. 2 denotes a non-reference frame. In general, an I picture is encoded only by its encoded data, and on the contrary, the information amount thereof is larger than that of encoded data of a P picture or a B picture. In order to suppress the amount of information of the encoded data and improve the compression rate, the number of I frames in a group is generally smaller than the number of P frames or B frames.

However, when the number of I frames is large, the image quality of a decoded image obtained by decoding encoded data and the resistance to an error mixed in the encoded data tend to be improved.

In the present embodiment, predetermined reference image information is set in advance in the moving image encoding unit 105, and the encoding process is repeated for each group using the predetermined reference image information without being limited to instructions such as an operation. The reference image information is information indicating the type of each frame in a group consisting of a predetermined number of frames. The reference image information includes reference destination frame information indicating a frame to be referred to, among frames of which the type is a P frame or a B frame. Therefore, if no special instruction is given, the non-reference frame is repeated at a predetermined repetition period TI.

Next, assume that an operation signal instructing to capture a still image (still image capture 1) is input from the operation unit 101 at time t2 in accordance with a user operation. At this time, the imaging control unit 121 determines that the imaging of the still image is instructed. When a period from an instruction time point, which is a time point at which the still image is instructed to be captured, to the next IDR frame (hereinafter referred to as an IDR frame standby period) is equal to or shorter than a predetermined period, the imaging control unit 121 causes the second imaging unit 103 to image the still image at the IDR frame time point.

on the other hand, the moving picture coding unit 105 specifies the order of the frames to be processed at the time point in the repetition period of the frames of the reference picture information set by the unit. The moving picture coding unit 105 can determine the IDR frame standby period T2 based on a specific order within the repetition period. The moving picture coding unit 105 notifies the imaging control unit 121 of the identified IDR frame standby period T2. When the frame type to be processed at this point in time is an IDR frame, the moving picture coding unit 105 outputs IDR information indicating that the frame type is an IDR frame to the control unit 104.

When the IDR frame waiting period T2 input from the moving picture coding unit 105 exceeds the threshold Δ T of the IDR frame waiting period preset in the present unit, the imaging control unit 121 causes the second imaging unit 103 to record the image of the frame at that point in time as a still image without waiting for the next IDR frame. At this time, the imaging control unit 121 outputs an imaging instruction signal to the second imaging unit 103. When the shooting instruction signal is input from the shooting control unit 121, the second shooting unit 103 generates second image data representing a second image, and outputs the generated second image data to the still image encoding unit 106. The still image encoding unit 106 generates second encoded data by performing encoding processing on the second image data input from the second imaging unit 103, and records the generated second encoded data in the recording unit 107.

The threshold Δ T of the IDR frame standby period may be within an allowable delay time (for example, 0.2 to 1.0 second) from an instruction to capture a still image to a real image.

Then, the imaging control unit 121 sets the frame a at this time point as an IDR frame, and causes the dynamic image encoding unit 105 to encode the second image data of the frame. At this time, the imaging control unit 121 outputs an IDR insertion request signal to the moving picture encoding unit 105. The IDR insertion request signal is a signal indicating encoding as an IDR frame.

When an IDR insertion request signal is input from the imaging control unit 121, the moving picture encoding unit 105 identifies a frame to be encoded as an IDR frame at that point in time, and encodes the identified frame as the IDR frame. The moving picture coding unit 105 associates first coded data obtained by coding the picture of the frame with second reference picture information including the frame and stores the associated data in the recording unit 107. The moving picture coding unit 105 may set the second reference picture information for each of a plurality of frame numbers in advance. The number of frames of the second reference picture information is longer than the threshold Δ T of the IDR standby period and corresponds to the number of frames in the IDR standby period shorter than the repetition period TI.

The moving picture coding unit 105 specifies the number of frames until the frame immediately before the IDR frame of the next repetition period, selects the second reference picture information of the specified number of frames, and stores the code indicating the selected second reference picture information in the recording unit 107 as a part of the first coded data, based on the frame to which the IDR insertion request signal is input. The moving image encoding unit 105 performs encoding processing using a prediction method corresponding to the type of the frame indicated by the selected second reference image information from among the frames in the current repetition period at that point in time. When the prediction method is inter prediction, the moving image encoding unit 105 performs inter prediction using a reference destination frame set for a target frame indicating the second reference image information. Further, the moving picture coding unit 105 may restart the coding process from the next IDR frame using predetermined reference picture information having a predetermined repetition period.

Next, it is assumed that an operation signal instructing to capture a still image (still image capture 2) is input from the operation unit 101 at time t3 in accordance with a user operation. At this time, the imaging control unit 121 determines that the imaging of the still image is instructed. The imaging control unit 121 causes the second imaging unit 103 to image the still image at the time point of the next IDR frame when the period from the instruction time point, which is the time point at which the still image is instructed to be imaged, to the next IDR frame is equal to or shorter than a predetermined period.

More specifically, when the IDR frame waiting period T3 input from the moving picture coding unit 105 is equal to or less than the predetermined IDR frame waiting period Δ T preset in the present unit, the imaging control unit 121 waits until the frame to be processed becomes the next IDR frame. Here, the imaging control unit 121 sets a waiting state of the IDR information from the moving picture coding unit 105. While the standby state of the IDR information is set, the imaging control unit 121 waits for the input of the IDR information from the moving picture encoding unit 105. When IDR information is input from the moving picture coding unit 105, the imaging control unit 121 releases the waiting state of the set IDR information. After that, the photographing control section 121 outputs a photographing instruction signal for instructing photographing of the still image to the second photographing section 103.

when the imaging instruction signal is input from the imaging control unit 121, the second imaging unit 103 captures a second image after the IDR frame standby period T3 has elapsed. The second imaging unit 103 generates second image data representing the second image to be imaged, and outputs the generated second image data to the still image encoding unit 106. The still image encoding unit 106 generates second encoded data by performing encoding processing on the second image data input from the second imaging unit 103, and records the generated second encoded data in the recording unit 107.

Next, when an operation signal instructing to stop the moving image capturing is input from the operation unit 101 at time t4 in accordance with a user operation, the imaging control unit 121 outputs an imaging control signal instructing to stop the moving image capturing to the second imaging unit 103. When the shooting control signal is input from the shooting control unit 121, the second shooting unit 103 stops shooting of the second image, which is a moving image, and stops output of the second image data representing the second image. The imaging control unit 121 outputs an encoding control signal instructing the moving image encoding unit 105 to stop the encoding process. When the encoding control signal is input from the imaging control unit 121, the moving image encoding unit 105 stops encoding processing for the second image data.

At time t5, when the user operation on operation unit 101 instructs the stop of power supply (power off), control unit 104 stops the operation of each unit, and the power supply unit stops the supply of power to each unit including control unit 104, thereby stopping the operation of each unit.

(image recording processing)

next, the image recording process according to the present embodiment will be described.

Fig. 3 is a flowchart showing an example of the image recording process according to the present embodiment.

(step S102) the imaging control unit 121 determines whether or not to instruct the start of imaging of the moving image based on the operation signal input from the operation unit 101. When the start of shooting is instructed (yes in step S102), the process proceeds to step S104. When the start of shooting is not instructed (no at step S102), the process at step S102 is repeated.

(step S104) the imaging control unit 121 determines whether or not to instruct imaging of a still image based on the operation signal input from the operation unit 101. When shooting is instructed (yes in step S104), the process proceeds to step S106. When photographing is not instructed (no in step S104), the process proceeds to step S114.

(step S106) the imaging control unit 121 acquires the IDR frame standby period T from the first imaging unit 102. Thereafter, the process proceeds to step S108.

(step S108) the imaging control unit 121 determines whether or not the IDR frame standby period T is equal to or less than a threshold Δ T of a predetermined IDR frame standby period. When the IDR frame standby period T is equal to or less than the threshold Δ T (yes in step S108), the process proceeds to step S110.

when the IDR frame standby period T exceeds the threshold Δ T (no in step S108), the process proceeds to step S112.

(step S110) the imaging control unit 121 sets a waiting state of the IDR information from the moving picture coding unit 105. After that, the process proceeds to step S122.

(step S112) the imaging control unit 121 outputs the IDR insertion request signal to the moving picture encoding unit 105. When an IDR insertion request signal is input from the imaging control unit 121, the moving picture encoding unit 105 identifies a frame to be encoded as an IDR frame at that point in time. After that, the process proceeds to step S120.

at this point in time, (step S114) the imaging control unit 121 determines whether or not the standby state of the IDR information from the moving picture coding unit 105 is set. If the standby state is set (yes at step S114), the process proceeds to step S116. If the waiting state is not set (no in step S114), the process proceeds to step S122.

(step S116) the imaging control unit 121 determines whether or not IDR information is input from the moving picture coding unit 105. When the IDR information is input (yes in step S116), the process proceeds to step S118. If the IDR information is not input (no in step S116), the process proceeds to step S122.

(step S118) the imaging control unit 121 releases the waiting state (clear) of the set IDR information. In the state where the waiting state is released, the imaging control unit 121 may discard reception of the IDR information from the moving picture coding unit 105 without waiting. After that, the process proceeds to step S120.

(step S120) the image pickup control unit 121 outputs an image pickup control signal indicating image pickup of the still image to the second image pickup unit 103. The second imaging unit 103 images a second image. The still image encoding unit 106 encodes second image data representing a second image, and records second encoded data obtained by the encoding in the recording unit 107. After that, the process proceeds to step S122.

(step S122) the moving image encoding unit 105 encodes the first encoded data input from the first imaging unit 102, and records the first encoded data obtained by the encoding in the recording unit 107. Thereafter, the frame to be processed is shifted to the next frame, and the process proceeds to step S104.

In the examples shown in fig. 2 and 3, the moving image encoding unit 105 autonomously notifies the imaging control unit 121 of the IDR frame standby period, but the present invention is not limited to this. When an operation signal instructing to capture a still image is input, the capture control unit 121 may output a first polling signal indicating a polling during the IDR frame standby period to the moving image encoding unit 105.

The moving picture coding unit 105 notifies the imaging control unit 121 of the IDR frame standby period T determined at this point in time as a response to the first polling signal.

(image display)

Next, an example of displaying an image captured by the image recording apparatus 10 will be described.

Fig. 4 is a diagram showing an example of display of an image captured by the image recording apparatus 10 according to the present embodiment.

When an operation signal for instructing display of a moving image is input, the display control unit 122 causes the display unit 112 to display a predetermined viewer Vw01, and causes the image decoding unit 111 to read first encoded data representing the moving image instructed by the operation signal from the recording unit 107. The image decoding unit 111 reads second encoded data representing a still image stored in association with the first encoded data and time information representing the time associated with the first encoded data. The time information may be a time based on the time of the beginning of the moving picture represented by the first encoded data, that is, a playback time indicated as an elapsed time from the beginning. The image decoding unit 111 decodes the read second encoded data to generate second image data representing a still image. The generated second image data and time information are output to the display control unit 122. The image decoding unit 111 uses a decoding method corresponding to the encoding method used in encoding when decoding the second encoded data.

The viewer Vw01 mostly occupies the display area for displaying the moving image MI01, and the progress bar Sb01 is displayed at the lower end portion not overlapping with the display area. The progress bar Sb01 visually represents the playing status of the video, and is a screen member for indicating the playing start point by an operation. In the example shown in fig. 4, the longitudinal direction of the progress bar Sb01 is oriented in the horizontal direction, and the length thereof indicates the playback time of the entire moving image. The progress bar Sb01 has an indicator Pt 01. The display control unit 122 displays the pointer Pt01 at a position corresponding to the playback time point notified by the image decoding unit 111. The distance from the left end of the progress bar Sb01 to the display position of the indicator Pt01 represents the elapsed time from the beginning of the video to the point of play time. Then, the display control unit 122 changes the display position of the indicator Pt01 at the position indicated by the operation signal input from the operation unit 101, and instructs to play the video from the play time point corresponding to the changed display position.

the display control unit 122 generates a thumbnail image by reducing a second image indicated by the second image data input from the image decoding unit 111 to a predetermined size in the playback time of the moving image indicated by the progressive bar Sb 01. The display control unit 122 displays the generated thumbnail image at a position corresponding to the time indicated by the input time information. The display positions of the thumbnails SI01, SI02, and SI03 shown in fig. 4 are positions corresponding to the times at which the respective second images were captured. The first image of the frame at that time represents an icon common to the second image.

When sensing that any of the displayed thumbnails is pressed, the display control unit 122 causes the image decoding unit 111 to start decoding of the first encoded data from the playback time point corresponding to the position of the thumbnail. Here, the operation signal including the coordinates in the display area is input from the operation unit 101. The image decoding unit 111 sequentially outputs first image data obtained by decoding the first encoded data to the display unit 112. The display unit 112 displays a first image (moving image) indicated by the first image data input from the image decoding unit 111.

Therefore, when the user instructs a thumbnail by an operation, the first image, which is a moving image, can be displayed on the display unit 112 from the playback time point corresponding to the display position of the instructed thumbnail. A frame of the first image at a point of time when the second image which is the base of the thumbnail is encoded and recorded becomes an IDR frame. Therefore, the image decoding unit 111 can decode the first encoded data after the IDR frame is started. Here, the image decoding unit 111 sets the reference image information and the second reference image information in advance, and specifies the second reference image information indicated by the coding included in the first coded data. The image decoding unit 111 decodes the first encoded data for each frame by referring to the specific second reference image information.

In this case, the image decoding unit 111 can decode the first encoded data from the IDR frame without using encoded data of a frame that is past the IDR frame. The image decoding unit 111 may include a data buffer for temporarily storing a part of the first encoded data to be processed when decoding the first encoded data of the inter prediction frame. In the present embodiment, when the IDR frame waiting period is equal to or less than the predetermined IDR frame waiting period, the frame waiting period is set to be the next IDR frame until the frame to be processed becomes the next IDR frame. Therefore, an increase in the capacity required as a data buffer can be suppressed. The decoding of the first encoded data is not limited to the image decoding unit 111, and can be performed by another device. In other devices, the frequency of an IDR frame or other I frame is not necessarily considered when setting the capacity required as a data buffer. Therefore, according to the present embodiment, the risk of exceeding the capacity of the data buffer in another device can be reduced. When the IDR frame waiting period exceeds the predetermined IDR frame waiting period, the decoding of the first encoded data can be started immediately without waiting for the next IDR frame.

(modification example)

In the above description, the case where shooting of a still image is instructed by a user operation is taken as an example, but the invention is not limited thereto. As shown in fig. 5, the control unit 104 may include an imaging determination unit 124.

the imaging determination unit 124 calculates an evaluation value indicating the acceptability of the imaging target of the first image indicated by the first image data input from the first imaging unit 102, and determines whether or not to image the second image based on whether or not the calculated evaluation value is higher than a threshold value of a predetermined evaluation value. More specifically, the imaging determination unit 124 can calculate the evaluation value using a known pattern recognition process.

in the pattern recognition processing, for example, a deep learning model, a mechanical learning model such as Adaboost, or the like can be used. The deep learning model is a mathematical model that constitutes a neural network having an input layer, 2 or more intermediate layers, and an output layer. The input layer, each intermediate layer, and the output layer have a plurality of input terminals, nodes (nodes), and output terminals, respectively. Each input terminal outputs the input value as an output value to each node of the first intermediate layer. Each node in the intermediate layer calculates a composite value by linearly combining input values input from nodes in the lower layer, and calculates an output value using a predetermined activation function with respect to the calculated composite value. However, the input terminal is regarded as a node of layer 0. Each node of the intermediate layer outputs the calculated output value to each node of the higher-level layer. However, the output terminal is regarded as a node of the uppermost intermediate layer. Each output terminal outputs, as an output value, a composite value obtained by linearly combining output values from each node of the uppermost intermediate layer as input values. As the activation function, for example, a sigmoid function, a hyperbolic tangent (tanh: hyperbolic tangent) function, or the like can be used. Therefore, the deep learning model is configured to include, as parameters, a linear combination coefficient for linearly combining the input values at each node in the intermediate layer, a parameter of an activation function of each node, and a linear combination coefficient for linearly combining the input values at the output end. In the following description, parameters constituting the machine learning model are referred to as model parameters.

The imaging determination unit 124 executes one or both of the first evaluation process and the second evaluation process described below when calculating the evaluation value.

In the first evaluation process, the photographing determination unit 124 calculates a first photographing evaluation value using a first mechanical learning model based on the first image data. The first imaging evaluation value is a value indicating the possibility that the main subject indicated by the evaluation image is the specific recognition target object. The candidate of the recognition target object is, for example, a human, an animal, a flower, a ball used in ball game, or the like. The candidate of the recognition target object may be a face of a person having a specific expression (e.g., smiling). The model parameters constituting the first mechanical learning model are generated (learned) in advance before the first evaluation process is performed, and are set in the imaging determination unit 124. As teacher data used in learning, image data representing an image showing a subject is used. The learning of the model parameters of the first machine learning model is performed by a model learning unit (not shown) (offline processing). The image recording apparatus 10 may or may not include a model learning unit. The imaging determination unit 124 may acquire the model parameters of the first mechanical learning model from an external device (for example, a server device) independent of the image recording apparatus 10.

The learning of the model parameters of the first machine learning model includes the following steps S11 and S12 (not shown).

(step S11) the model learning unit calculates a first imaging evaluation value for each candidate of a known recognition target using a first mechanical learning model with respect to teacher data representing an image of a known subject. In the case where a deep learning model is used as the first mechanical learning model, the model learning section determines the input values to the input terminals based on, for example, a signal value per pixel represented by teacher data. The model learning unit may use a signal value for each pixel as an input value, or may use an element value constituting an image feature amount obtained based on the signal value as an input value. As the image feature amount, for example, an edge direction per pixel, a gradient of a signal value per pixel, or the like can be used. The model learning unit specifies, as the first imaging evaluation value of the candidate of the recognition target corresponding to the output terminal, the output value from each output terminal calculated by using the deep learning model with respect to the specified input value.

The model learning unit updates the model parameters in sequence (step S12) until the first imaging evaluation value calculated for the recognition target candidate of the same type as the known subject indicated by the teacher data becomes a predetermined first positive value (for example, 1), and until the first imaging evaluation value obtained for the recognition target candidate of a different type from the known subject becomes a predetermined second positive value or less. As the predetermined second positive value, a positive real number substantially close to zero compared to the first positive value is used. Thus, the larger the value of the first imaging evaluation value for the specific recognition target candidate calculated using the deep learning model is, the higher the possibility that the object is the recognition target thereof is. In the learning of the model parameters, a sufficiently large number (for example, several thousands to several tens of thousands) of different teacher data are used.

the imaging determination unit 124 calculates a first imaging evaluation value for the evaluation image using the first mechanical learning model (online processing). When the deep learning model is used as the first mechanical learning model, the imaging determination unit 124 determines the input value to each input terminal based on the signal value representing each pixel of the evaluation image by the same method as in the learning. The imaging determination unit 124 calculates an output value from an output end of the deep learning model with respect to the input value as a first imaging evaluation value of a candidate of the recognition target corresponding to the output end.

The step of calculating the first imaging evaluation value may include a search step of searching for an area indicating a candidate of the recognition target object and a shape of the candidate. The search process includes, for example, the following processes: a first imaging evaluation value is calculated using a first mechanical learning model for an evaluation image corrected by performing linear conversion using a linear conversion coefficient for each of a plurality of linear conversion coefficients set in advance, and the linear conversion coefficient to which the maximum first imaging evaluation value is given is specified. The linear conversion coefficient depends on the area and shape of the candidate representing the recognition target. Therefore, the area and the shape of the recognition target object are specified by the specified linear conversion coefficient.

When the second evaluation process is not performed, the imaging determination unit 124 specifies, as an identification target, the identification target candidate having the largest first imaging evaluation value among the first imaging evaluation values calculated for the identification target candidates, and specifies the first imaging evaluation value as the imaging evaluation value.

in the second evaluation process, the imaging determination unit 124 calculates an evaluation value indicating how well the composition of the evaluation image is. The elements that affect the acceptability or non-acceptability of the composition are, for example, the position of a region indicating a main subject with respect to the entire evaluation image including the background, the size thereof, the color of the main subject and the background, the shade thereof, and the like. The model learning unit learns in advance the model parameters of the second machine learning model used in the second evaluation process, and sets the model parameters in the imaging determination unit 124. As teacher data used in learning, image data representing a main subject and an image representing a background is used.

the learning of the model parameters of the second machine learning model includes the following steps S21 and S22 (not shown).

(step S21) the model learning unit calculates a second imaging evaluation value using a second mechanical learning model with respect to the teacher data. In the case where a deep learning model is used as the second mechanical learning model, the model learning section determines the input values to the input terminals based on the signal value per pixel represented by the teacher data. The model learning unit determines an output value from the output terminal calculated using the deep learning model with respect to the determined input value as the second photographing evaluation value.

(step S22) the model learning unit updates the model parameters constituting the second machine learning model so that the square value of the difference between the second imaging evaluation value and the point given to the image represented by the teacher data is equal to or less than a predetermined square value. The predetermined square value may be a positive value sufficiently close to 0. The point is, for example, a value indicating whether or not the human subjective judgment is acceptable with respect to the image. The point number may be 1 when it is determined to be good, or may be a 2-level value of 0 when it is not determined to be good. The point number may be a value of a plurality of levels of 3 or more, which is higher as the point number is better.

The imaging determination unit 124 calculates a second imaging evaluation value using the second mechanical learning model for the evaluation image (online processing). When the deep learning model is used as the second mechanical learning model, the imaging determination unit 124 determines the input value to each input terminal based on the signal value representing each pixel of the evaluation image by the same method as in the learning. The imaging determination unit 124 calculates an output value from an output end of the deep learning model with respect to the input value as a second imaging evaluation value.

In the above example, the case where the second evaluation process is performed as the second mechanical learning model independently of the first evaluation process, and the step of calculating the second imaging evaluation value with respect to the evaluation image using the deep learning model is included as the second mechanical learning model is exemplified, but the present invention is not limited thereto. The method may further comprise the steps of: the imaging determination unit 124 calculates a second imaging evaluation value based on a region (hereinafter, referred to as an object region) that represents the recognition object, which is the processing result of the first evaluation processing, without using the second mechanical learning model as the second evaluation processing. The imaging determination unit 124 determines, for example, the second imaging evaluation value such that the second imaging evaluation value is higher as the ratio of the portion included in the predetermined second evaluation region in the target region is larger. However, as the second evaluation area, the evaluation area is shared with the center of gravity and has a similar shape, and an area having a smaller diameter on one side (for example, 0.2 to 0.6 times the one side of the evaluation area) than the evaluation area is set in advance in the imaging determination unit 124. Thus, the second imaging evaluation value that becomes higher as the preset object region of the object is located within the second evaluation region (for example, the center portion of the evaluation region) of the predetermined size is calculated. The second imaging evaluation value calculated in this way is higher as the person as the recognition target object is enlarged in the center of the evaluation area, for example.

In addition, the imaging determination unit 124 may set second imaging evaluation data indicating the second imaging evaluation value in advance for each of the type of the recognition target object, the size of the target object area, and the position. The imaging determination unit 124 may specify the second imaging evaluation value corresponding to the group of the type of the recognition target object, the size of the target object region, and the position obtained in the first evaluation process with reference to the set second imaging evaluation data.

when the first evaluation process is not performed, the imaging determination unit 124 determines the calculated second imaging evaluation value as the imaging evaluation value.

When both the first evaluation process and the second evaluation process are performed, the imaging determination unit 124 multiplies the first imaging evaluation value and the second imaging evaluation value by the respective predetermined weight coefficients, and calculates the sum of multiplication values obtained by the multiplication as the imaging evaluation value.

in the above description, the threshold Δ T during the IDR frame standby period is assumed to be constant, but may be variable.

The imaging control unit 121 may determine the threshold Δ T such that the threshold Δ T is shorter as the motion amount of the moving image of the first image of the frame at that time point is larger. The imaging control unit 121 sets in advance standby period control data indicating a threshold value Δ T in association with the amount of motion, and can specify the threshold value Δ T in association with the amount of motion at that point in time.

The imaging control unit 121 can use, as an index value of the motion amount of the first image, a representative value of the magnitude of the motion vector calculated for each block by the moving image encoding unit 105 in the encoding step. As the representative value, any of an average value, a mode value, and a maximum value among blocks in a frame can be used.

When detecting the motion of the first image, the moving image encoding unit 105 performs block matching between blocks in the reference image for each object block obtained by dividing the first image of the current frame. The moving image encoding unit 105 detects the pattern indicated by the target block and the block indicating the most approximate pattern as corresponding blocks. In block matching, SAD (Sum of Absolute Differences) can be used as an evaluation value indicating the degree of approximation of a pattern, for example. SAD is an index value representing a higher degree of approximation as its value is smaller. Then, the moving image encoding unit 105 determines, as a motion vector, a displacement (offset) from the position of the corresponding block in the reference image detected for each object block. The current frame and the frame of the reference image (hereinafter referred to as a reference frame) are not necessarily adjacent to each other, and may be separated by 2 frames or more. The moving image encoding unit 105 normalizes the absolute value of the motion vector for each block at a frame interval between the current frame and the reference frame, and outputs the normalized value to the imaging control unit 121 as the magnitude of the motion vector.

As described above, the case where the imaging control unit 121 acquires the motion vector calculated for each block by the moving image encoding unit 105 in the encoding step and uses the representative value of the magnitude of the motion vector as the index value of the motion amount of the first image is exemplified, but the present invention is not limited thereto. The control unit 104 may further include a moving image analysis unit (not shown) that analyzes the motion (optical flow) of the first image indicated by the first image data input from the first imaging unit 102. For example, as described above, the moving image analysis unit detects, for each object block obtained by dividing the first image of the current frame, a first image of a past frame (for example, a frame immediately preceding the current frame) as a reference image, performs block matching between blocks in the reference image, and detects, as a corresponding block, a pattern indicated by the object block and a block in the reference image indicating the most similar pattern. The moving image analysis unit determines the displacement from the position of the corresponding block detected for each object block as a motion vector, and outputs the magnitude of the motion vector to the imaging control unit 121. The imaging control unit 121 obtains an index value of the motion amount of the first image based on the size of the motion vector determined by the moving image analysis unit, instead of the moving image encoding unit 105. In the encoding process, inter prediction is not performed on the I picture, and therefore, no motion vector is acquired. However, by providing the moving image analysis unit, even when the frame to be encoded is an I picture, the amount of motion of the first image can be quantified without depending on the moving image encoding unit 105.

In addition, when the image recording apparatus 10 includes the image capturing determination unit 124, the image capturing control unit 121 may calculate a representative value of the magnitude of a motion vector between blocks overlapping an identification region that is a region indicating a main identification target object, as an index value of the motion amount of the first image. The size of the motion vector corresponds to the amount of displacement of the object appearing in the recognition area from the reference image to the current frame. The imaging determination unit 124 can identify the recognition area in the step of performing the first evaluation process. The imaging determination unit 124 outputs the specific identification area to the imaging control unit 121, and the imaging control unit 121 can use information of the identification area when calculating the index value. Thus, the imaging control unit 121 can determine the threshold Δ T based on the amount of movement of the main object.

Further, the photographing control section 121 may determine the threshold value Δ T so that the threshold value Δ T becomes shorter as the movement amount of the image recording apparatus 10 itself becomes larger. The imaging control unit 121 sets in advance standby period control data indicating a threshold value Δ T in association with the amount of motion, and can specify the threshold value Δ T in association with the amount of motion at that point in time.

the image recording apparatus 10 may further include an acceleration detection unit (not shown) for detecting the amount of movement. The acceleration detection unit is, for example, a 3-axis acceleration sensor. The acceleration detection unit detects the acceleration in each sensitivity axis direction, and outputs the detected acceleration to the imaging control unit 121. Then, the imaging control unit 121 calculates a weighted moving average of the accelerations in the sensitivity axis directions, which is input from the acceleration detection unit, as a gravity component. The imaging control unit 121 calculates a motion component of the acceleration by subtracting the gravity component from the acceleration input from the acceleration detection unit, and calculates the square root of the sum of squares of the sensitivity axes of the velocity components obtained by integrating the calculated motion component of the acceleration as time, as the motion amount. Thereby, the threshold Δ T is determined based on the movement amount of the image recording apparatus 10.

In the example shown in fig. 1 and 5, the first image capturing unit 102 and the second image capturing unit 103 are separate, but the image recording apparatus 10 may include at least one image capturing unit (not shown). One imaging unit may be used in common for imaging a first image as a moving image and a second image as a still image.

The moving image encoding unit 105 encodes first image data representing an image captured by a common imaging unit as a moving image, and the still image encoding unit 106 encodes second image data representing the image captured by the imaging unit as a still image.

the image recording apparatus 10 may further include a sound collection unit, a speech encoding unit, a speech decoding unit, and a playback unit (not shown). The voice acquisition unit acquires voice of a voice arriving at the home unit, and outputs voice data representing the voice acquired by the voice acquisition unit to the voice encoding unit. The voice encoding unit encodes the voice data input from the voice acquisition unit to generate voice encoded data. The speech encoding unit may record the speech encoded data in the recording unit 107 in association with the first encoded data. When the control unit 104 outputs a control signal indicating playback of the audio corresponding to the first image in the image decoding unit to the audio decoding unit, the audio decoding unit reads the audio encoded data instructed by the control signal from the recording unit. The audio decoding unit performs decoding processing on the read encoded audio data to generate audio data, and outputs the generated audio data to the playback unit. The playing section plays the voice based on the voice data input by the voice decoding section.

As described above, the image recording apparatus 10 according to the present embodiment includes: a moving image encoding unit 105 for encoding an image of each frame forming a moving image; and a control unit 104. The sequence of frames forming a dynamic image includes: the moving image encoding unit 105 encodes an intra-frame prediction frame by performing intra-frame prediction, and encodes an inter-frame prediction frame by performing inter-frame prediction by referring to an image of another frame as a reference image. The sequence of frames includes, in the intra-frame prediction frames, non-reference frames that are not referred to in inter-frame prediction of the past inter-frame prediction frames, at every predetermined cycle. The control unit 104 waits until the next non-reference frame when a waiting period from an instruction time point for instructing the shooting of the still image to the next non-reference frame is equal to or less than a predetermined time, and encodes the moving image encoding unit using the image of the frame as the non-reference frame when the waiting period exceeds the predetermined time.

According to this configuration, when the standby period is equal to or shorter than the predetermined time, the frame indicating the time point is not encoded as the non-reference frame until the next non-reference frame. Therefore, compared to the case where encoding is performed uniformly as a non-reference frame, an increase in encoded data accompanying an increase in non-reference frames can be suppressed. Therefore, the encoded data exceeds the buffer capacity in the decoding process, and the risk of not normally playing the image can be reduced. The first encoded data and the second encoded data recorded by the image recording apparatus 10 can be decoded in other electronic devices, but this risk is highly likely to occur in other electronic devices having a buffer capacity determined regardless of the design of the image recording apparatus 10.

The control unit 104 may shorten the predetermined time as the motion amount of the moving image increases.

In general, as the motion amount of a moving image increases, the time from the time point when shooting is instructed to the icon coded as a non-reference frame increases, and therefore the time allowed as a standby period decreases. By shortening the predetermined time in conjunction with the allowable standby period, it is possible to further record an image of the non-reference frame as intended.

the moving image encoding unit 105 specifies a motion vector for each block in the inter prediction frame, and the control unit 104 specifies the motion amount based on the motion vector.

According to this configuration, the motion vector calculated in the step of encoding a moving image can be used when determining the amount of motion, and therefore an increase in the amount of processing can be suppressed.

The control unit 104 may recognize a predetermined object from the image of each frame and determine a representative value of the motion vector for each block in the display area of the object as the motion amount.

According to this configuration, since the motion vector in the display region of the predetermined object appearing in the moving image is used when the motion amount is determined, the predetermined time is determined based on the motion of the object.

The image recording apparatus 10 may further include an acceleration detection unit that detects acceleration. The control unit 104 determines the amount of exercise of the present apparatus based on the acceleration detected by the acceleration detection unit, and shortens the predetermined time as the amount of exercise of the present apparatus increases.

As the amount of movement of the image recording apparatus 10 during shooting increases, the amount of change in the icons of the moving images being shot increases, and therefore the time allowed as the standby period decreases. By shortening the predetermined time in conjunction with the standby period, it is possible to further record an image of a non-reference frame as intended.

The control unit 104 calculates an evaluation value indicating the acceptability of the subject to be captured as a moving image, and determines whether or not to capture a still image based on the calculated evaluation value.

According to this configuration, the quality of the subject to be captured is quantitatively evaluated based on the moving image, and whether or not the still image needs to be captured is determined based on the evaluated quality. Therefore, as compared with the case where the encoding is uniformly performed as the non-reference frame based on the determination result, the increase of the encoded data accompanying the increase of the non-reference frame can be suppressed.

In the above-described embodiment, in the size determination with respect to the threshold value (for example, the threshold value Δ T), it is arbitrary to perform the determination including the threshold value as "above" or "below", or to perform the determination not including the threshold value as "larger", "smaller", "exceeding", "not exceeding", "larger", "smaller", "insufficient", or the like. For example, "above" may be replaced with "larger", "exceeding", or "greater than", and "below" may be replaced with "smaller", "not exceeding", "less than", or "insufficient".

Further, a part of the image recording apparatus 10, for example, the control unit 104, the moving image encoding unit 105, the still image encoding unit 106, and the image decoding unit 111, or any combination thereof may be configured as a computer including a processor such as a CPU, and may implement these functions by executing processing instructed by a command described in a program stored in advance in a storage medium. In this case, the program for realizing the control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system.

Further, a part of the image recording apparatus 10 in the above-described embodiment may be configured to include a Large Scale Integrated Circuit (LSI), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or the like. Each functional block of a part of the image recording apparatus 10 may be individually processed, or a part or all of the functional blocks may be integrated and processed. The method of integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when a technology for forming an integrated circuit instead of an LSI or the like has emerged due to the advancement of semiconductor technology, an integrated circuit using the technology may be used.

The embodiments of the present invention are described in detail with reference to the drawings, and specific configurations are not limited to the above embodiments, and include designs and the like that do not depart from the scope of the present invention. The respective configurations described in the above embodiments can be combined arbitrarily.

Description of the reference numerals

10 … image recording means; 101 … operation part; 102 … a first imaging unit; 103 … a second imaging unit; 104 … control section; 105 … a moving picture encoding unit; 106 … a still image encoding unit; 107 … recording part; 111 … image decoding unit; 112 … display part; 113 … a communication section; 121 … imaging control part; 122 … display control unit; 124 … image capture determination unit.