阅读说明：本技术 文件生成设备、文件生成方法、文件再现设备、文件再现方法和程序 (File generation device, file generation method, file reproduction device, file reproduction method, and program ) 是由 船元大辅 伊藤亮吾 矶部幸雄 石坂敏弥 松井裕司 山田崇晴 于 2020-03-16 设计创作，主要内容包括：本技术关于能够实现存储在文件中的图像与文件之外的外部数据之间的关联的文件生成设备、文件生成方法、文件再现设备、文件再现方法和程序。根据本技术,文件控制单元生成关联型高效图像文件格式(HEIF)文件,在关联型高效图像文件格式文件中HEIF文件中符合HEIF的图像与指定HEIF文件之外的外部数据的特定信息被关联并存储,外部数据要与图像相关联。文件控制单元还再现关联型HEIF文件。本技术能够应用于HEIF文件的生成或再现。(The present technology pertains to a file generation device, a file generation method, a file reproduction device, a file reproduction method, and a program that enable association between an image stored in a file and external data other than the file. According to the present technology, a file control unit generates an association-type high-efficiency image file format (HEIF) file in which an image conforming to HEIF in a HEIF file is associated with specific information specifying external data other than the HEIF file and stored, the external data being to be associated with the image. The file control unit also reproduces the associated type HEIF file. The present technology can be applied to the generation or reproduction of the HEIF file.)

1. A file generating apparatus comprising:

a file control unit that generates an associated type high-efficiency image file format (HEIF) file in which an image complying with HEIF in an HEIF file and specific information specifying external data other than the HEIF file, the external data being associated with the image, are stored in association with each other.

2. The file generating device according to claim 1, wherein the file controlling unit generates an association-type HEIF file in which association information associating the image with specific information of the external data is stored.

3. The file generating apparatus according to claim 2, wherein the file controlling unit stores the association information in which an item ID specifying the image corresponds to the specific information, in the association-type HEIF file.

4. The file generating apparatus according to claim 3, wherein the file controlling unit stores the association information in a meta box or an mdat box of the association-type HEIF file.

5. The file generating apparatus according to claim 2, wherein said file control unit

Storing the specific information in an mdat box of the associative type HEIF file, an

Storing the association information in which the item ID specifying the image and the item ID specifying the specific information stored in the mdat box are associated in a meta box of the association-type HEIF file.

6. The file generating device according to claim 1, wherein the file control unit stores a track of specific information of the external data associated with each frame constituting a track of the image in an mdat box of the associated type HEIF file.

7. The file generating apparatus according to claim 6, wherein each frame constituting a track of the image and specific information constituting a track of the specific information are associated with each other by time point information on a time line or an order of arrangement in a track.

8. The file generation device according to claim 1, wherein the image and other images based on the image are stored in the association-type HEIF file.

9. A file generation method, comprising:

an associated type high-efficiency image file format (HEIF) file is generated in which an image conforming to HEIF in an HEIF file and specific information specifying external data other than the HEIF file, which is associated with the image, are stored in association with each other.

10. A program for causing a computer to function as follows

A file control unit that generates an associated type high-efficiency image file format (HEIF) file in which an image complying with HEIF in an HEIF file and specific information specifying external data other than the HEIF file, the external data being associated with the image, are stored in association with each other.

11. A file reproducing apparatus comprising:

a file control unit that reproduces an associated type high-efficiency image file format (HEIF) file in which an image complying with HEIF in an HEIF file and specific information specifying external data other than the HEIF file, the external data being associated with the image, are stored in association with each other.

12. The file reproducing apparatus according to claim 11, wherein said file control unit reads specific information of the external data associated with a predetermined image from the association type HEIF file, association information associating the image with the specific information of the external data being stored in the association type HEIF file.

13. The file reproducing apparatus according to claim 12, wherein the association information that associates the item ID specifying the image with the specific information is stored in the association-type HEIF file, and

the file control unit reads the specific information corresponding to an item ID of a predetermined image in the association information.

14. The file reproducing apparatus of claim 13,

wherein the associated information is stored in a meta box or an mdat box of the associated type HEIF file, an

The file control unit reads the specific information corresponding to an item ID of a predetermined image in the association information from the meta box or the mdat box.

15. The file reproducing apparatus of claim 12,

wherein the specific information is stored in an mdat box of the associated type HEIF file, an

Storing the association information in which the item ID specifying the image and the item ID specifying the specific information stored in the mdat box are associated with each other in a meta box of the association-type HEIF file, and

the file control unit reads specific information specified by an item ID of the specific information corresponding to an item ID of a predetermined image among the associated information stored in the meta box from the mdat box.

16. The file reproducing apparatus of claim 11,

wherein a track of specific information of the external data associated with each frame constituting a track of the image is stored in an mdat box of the association-type HEIF file, and

each frame constituting a track of the image and specific information constituting a track of the specific information are associated with each other by time point information on a time line or an order of arrangement in a track, an

The file control unit acquires, from the track of the specific information, time point information on a timeline of predetermined frames or the specific information of an order of arrangement of the predetermined frames in the track.

17. The file reproducing apparatus according to claim 11, wherein said image and other images based on said image are stored in said associated type HEIF file.

18. A file reproduction method comprising:

reproducing an associated type high efficiency image file format (HEIF) file in which an image complying with HEIF in an HEIF file and specific information specifying external data other than the HEIF file, the external data being associated with the image, are stored in association with each other.

19. A program for causing a computer to function as follows

A file control unit that reproduces an associated type high-efficiency image file format (HEIF) file in which an image complying with HEIF in an HEIF file and specific information specifying external data other than the HEIF file, the external data being associated with the image, are stored in association with each other.

Technical Field

The present technology relates to a file generating device, a file generating method, a file reproducing device, a file reproducing method, and a program, and particularly relates to a file generating device, a file generating method, a file reproducing device, a file reproducing method, and a program, for example, capable of associating an image stored in a file with external data other than the file.

Background

As a file format for efficiently storing an image, there is a high-efficiency image file format (HEIF).

Reference list

Non-patent document

Non-patent document 1: ISO/IEC 23008-12: 2017, Information technology: -High efficiency coding and media delivery in heterologous environments-Part 12: Image File Format

Disclosure of Invention

Problems to be solved by the invention

Regarding a HEIF file that conforms to a high-efficiency image file format (HEIF), it is convenient if an image stored in the HEIF file can be associated with external data other than the HEIF file.

The present technology is made in view of such a situation, and makes it possible to associate an image stored in a HEIF file with external data other than the HEIF file.

Solution to the problem

A file generating device or a first program according to the present technology is a file generating device including a file control unit that generates an associated type high-efficiency image file format (HEIF) file in which specific information of an image conforming to HEIF and external data other than a specified HEIF file in the HEIF file are stored in association with each other, the external data being associated with the image, or a program that causes a computer to function as such a file generating device.

A file generation method according to the present technology is a file generation method that includes generating an association-type high-efficiency image file format (HEIF) file in which an image that conforms to HEIF in the HEIF file and specific information that specifies external data other than the HEIF file, which is associated with the image, are stored in association with each other.

In the file generating device, the file generating method, and the first program according to the present technology, an associated type high-efficiency image file format (HEIF) file in which specific information of an image conforming to HEIF in the HEIF file and external data other than a specified HEIF file are stored in association with each other is generated, the external data being associated with the image.

The file reproducing device or the second program according to the present technology is a file reproducing device including a file control unit that reproduces an associated type high-efficiency image file format (HEIF) file in which specific information specifying an image conforming to HEIF and external data other than the HEIF file in the HEIF file are stored in association with each other, the external data being associated with the image, or a program that causes a computer to function as such a file reproducing device.

A file reproduction method according to the present technology is a file reproduction method including reproducing an associated type high-efficiency image file format (HEIF) file in which an image conforming to HEIF in the HEIF file and specific information specifying external data other than the HEIF file, which is associated with the image, are stored in association with each other.

In the file reproducing device, the file reproducing method, and the second program according to the present technology, an associated type high-efficiency image file format (HEIF) file in which an image conforming to HEIF in the HEIF file and specific information specifying external data other than the HEIF file are stored in association with each other is reproduced, the external data being associated with the image.

Note that the file generating device and the file reproducing device may be separate devices or internal blocks constituting one device.

Further, the first program and the second program may be provided by being recorded on a recording medium or transmitted via a transmission medium.

Drawings

Fig. 1 is a block diagram illustrating a configuration example of one embodiment of a digital camera to which the present technology is applied.

Fig. 2 is a diagram illustrating an example of a format of a Joint Photographic Experts Group (JPEG) file conforming to JPEG.

Fig. 3 is a diagram illustrating an example of an ISO base media file format.

Fig. 4 is a diagram illustrating an example of a format of a HEIF file conforming to HEIF.

Fig. 5 is a diagram illustrating an example of a format of a HEIF file in the form of an image item.

Fig. 6 is a diagram illustrating an example of an iprp cassette.

Fig. 7 is a diagram illustrating an example of a format of a HEIF file in the form of an image sequence.

Fig. 8 is a diagram illustrating an example of a trak box.

Fig. 9 is a diagram illustrating an example of a general collection file storing main images and thumbnail images.

Fig. 10 is a diagram illustrating an example of a first association-type aggregate file.

Fig. 11 is a diagram illustrating an example of a second association-type aggregate file.

Fig. 12 is a diagram illustrating an example of a third association-type aggregate file.

Fig. 13 is a diagram illustrating an example of a normal sequence file in which a track of a main image and a track of a thumbnail image of the main image are stored.

Fig. 14 is a diagram illustrating an example of an association-type sequence file.

Fig. 15 is a flowchart illustrating an outline of an example of the generation processing of generating the association-type HEIF file.

Fig. 16 is a flowchart illustrating an outline of an example of reproduction processing of reproducing the associated type HEIF file.

Fig. 17 is a flowchart illustrating an example of reproduction processing of reproducing an aggregate file.

Fig. 18 is a flowchart illustrating an example of the processing of reading the reproduction target image in step S32.

Fig. 19 is a flowchart illustrating a first example of the process of acquiring the reproduction target item ID in step S31.

Fig. 20 is a flowchart illustrating a second example of the process of acquiring the reproduction target item ID in step S31.

Fig. 21 is a flowchart illustrating an example of processing of acquiring uuid of specific information of a RAW file as a predetermined main image from a first association-type set file.

Fig. 22 is a flowchart illustrating an example of processing of acquiring uuid of specific information of a RAW file as a predetermined main image from the second association-type set file.

Fig. 23 is a flowchart illustrating an example of processing of acquiring uuid of the specific information of the RAW file as the predetermined main image from the third associated-type set file.

Fig. 24 is a flowchart illustrating an example of a process of acquiring a list of item IDs of a main image from a collection file.

Fig. 25 is a flowchart illustrating an example of processing of reproducing a thumbnail image for (a frame of) a main image of predetermined point-in-time information from a sequence file.

Fig. 26 is a flowchart illustrating an example of processing of acquiring the uuid of the specific information of the RAW file as (the frame of) the predetermined main image from the associated-type sequence file.

Fig. 27 is a diagram illustrating an example of storing uuid in the RAW file of the main image in the case of adopting the RAW file as external data and generating an association-type set file.

Fig. 28 is a diagram illustrating an example of storing uuid in the RAW file of the main image in the case of adopting the RAW file as external data and generating an association-type sequence file.

Fig. 29 is a diagram illustrating an example of storing uuid in the WAV file of the main image in the case of adopting the WAV file as external data and generating an association-type set file.

Fig. 30 is a diagram illustrating an example of storing uuid in the WAV file of the main image in the case of adopting the WAV file as external data and generating an association-type sequence file.

Fig. 31 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied.

Detailed Description

< one embodiment of digital Camera to which the present technology is applied >

Fig. 1 is a block diagram illustrating a configuration example of one embodiment of a digital camera to which the present technology is applied.

The digital camera 10 includes an optical system 11, an image sensor 12, a signal processing unit 13, a medium 14, interfaces 15 and 16, buttons/keys 17, a touch panel 18, a liquid crystal panel 19, a viewfinder 20, an interface 21, and the like.

The optical system 11 condenses light from an object on the image sensor 12.

The image sensor 12 receives light from the optical system 11 and performs imaging for photoelectric conversion to generate data of an image as an electric signal and supplies the data to the signal processing unit 13.

The signal processing unit 13 includes an optical system/image sensor control unit 41, an encoding control unit 42, a file control unit 43, a media control unit 44, an operation control unit 45, a display control unit 46, and a UI control unit 47.

The optical system/image sensor control unit 41 controls the optical system 11 and the image sensor 12, and supplies (data of) an image obtained by imaging performed according to the control to the encoding control unit 42.

The encoding control unit 42 supplies the image from the optical system/image sensor control unit 41 to the display control unit 46, and also encodes and supplies the image to the document control unit 43 as necessary. Further, the encoding control unit 42 decodes the image supplied from the file control unit 43 as necessary, and supplies the image to the display control unit 46.

The file control unit 43 generates a file storing the image supplied from the encoding control unit 42, and supplies the file to the medium control unit 44. Further, the file control unit 43 reproduces a file supplied from the media control unit 44, that is, reads data such as an image or the like stored in the file. For example, an image read from a document is supplied from the document control unit 43 to the encoding control unit 42.

The media control unit 44 controls file exchange with the media 14 and the interfaces 15 and 16. For example, the media control unit 44 causes a file from the file control unit 43 to be recorded on the medium 14 or transmitted from the interfaces 15 and 16. Further, the media control unit 44 reproduces a file from the medium 14 or causes a file to be received by the interfaces 15 and 16 and supplied to the file control unit 43.

The operation control unit 45 supplies an operation signal corresponding to an operation of the button/key 17 or the touch panel 18 by the user to a desired block.

The display control unit 46 performs display control and the like to supply the image and the like supplied from the encoding control unit 42 to the liquid crystal panel 19, the viewfinder 20, and the interface 21 and display the image and the like.

The UI control unit 47 manages User Interface (UI) control.

The medium 14 is a storage medium such as an SD card, for example. The interface 15 is an interface of a Local Area Network (LAN) such as WiFi (registered trademark) or ethernet (registered trademark), for example. The interface 16 is, for example, a Universal Serial Bus (USB) interface. When a command or other information is input to the digital camera 10, the buttons/keys 17 and the touch panel 18 are operated by the user. The touch panel 18 may be integrally formed with the liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display an image or the like supplied from the display control unit 46. The interface 21 is an interface for transmitting at least an image, such as a high-definition multimedia interface (HDMI) (registered trademark) or a Display Port (DP).

In the digital camera 10 configured as described above, the encoding control unit 42 generates, for example, a YUV image (hereinafter also referred to as a main image) having the same resolution (number of pixels) as that of the RAW image from an image (hereinafter also referred to as a RAW image) in the RAW data obtained by imaging by the image sensor 12. Also, the encoding control unit 42 generates, from the YUV main image, for display on the liquid crystal panel 19 or on the external display, for example, a YUV image having a resolution lower than that of the main image (hereinafter also referred to as a screen thumbnail image) as a first other image based on the main image, and generates, for list display, for example, a YUV image having a resolution lower than that of the screen thumbnail image (hereinafter also referred to as a thumbnail image) as a second other image based on the main image. The encode control unit 42 supplies the screen thumbnail image to the liquid crystal panel 19 via the display control unit 46, for example, and displays the screen thumbnail image as a so-called through-lens image. As the thumbnail image, for example, an image having a size of 320 pixels or less on the long side may be employed. The size (number of pixels) of the main image may be set to, for example, two hundred times or less the size (number of pixels) of a screen thumbnail image as the first other image based on the main image or a thumbnail image as the second other image based on the main image. Similarly, the size of the screen thumbnail image that is the first other image based on the main image may be set to, for example, two hundred times or less the size of the thumbnail image that is the second other image based on the main image. As the screen thumbnail image, for example, an image having a resolution of 4K or more can be employed. Further, as the screen thumbnail image, for example, a 4k (qfhd) or FHD image may be adopted according to the selection of the user. Also, images having the same resolution may be employed as the current image and the screen thumbnail image. In the case of adopting images having the same resolution as the current image and the screen thumbnail image, both the current image and the screen thumbnail image may be stored in the HEIF file, or the current image may be stored in the HEIF file without storing the screen thumbnail image therein. In the case where the current image is stored in the HEIF file without storing the screen thumbnail image therein, the current image may be resized and used as the screen thumbnail image.

Further, the encoding control unit 42 encodes the main image, the screen thumbnail image, and the thumbnail image (the main image, the screen thumbnail image, and the thumbnail image generated from the RAW image) corresponding to the RAW image as necessary, and supplies the encoded images to the file control unit 43 together with the RAW image.

The file control unit 43 generates a RAW file storing a RAW image, and generates an HEIF file or JPEG file storing a corresponding main image, screen thumbnail image, and thumbnail image (a main image, screen thumbnail image, and thumbnail image generated from the same RAW image), and supplies the HEIF file or JPEG file to the media control unit 44. The HEIF file is a file conforming to the high-efficiency image file format (HEIF), and the JPEG file is a file conforming to the Joint Photographic Experts Group (JPEG).

The medium control unit 44 causes the RAW file and the HEIF file or JPEG file from the file control unit 43 to be recorded on the medium 14, or causes the RAW file and the HEIF file or JPEG file to be transferred from the interface 15 or 16.

For example, which of the HEIF file and the JPEG file is to be generated in the file control unit 43 may be selected in accordance with an operation by the user. Further, as will be described later, there are an image item form and an image sequence form as the format of the HEIF file. For example, which of the image item form and the image sequence form is adopted may be selected according to the user's operation. Also, the file control unit 43 may perform interconversion between the HEIF file and the JPEG file according to the user's operation.

Also, at the time of generating the HEIF file, the file control unit 43 may associate internal data (data stored in the HEIF file) in the HEIF file associated with external data (data not stored in the HEIF file) other than the HEIF file with specific information specifying the external data, and store the internal data and the specific information in the HEIF file in association with each other. A HEIF file in which internal data and specific information of external data to be associated with the internal data are stored in association with each other is also referred to as an association type HEIF file. In the association-type HEIF file, the internal data and the specific information may be stored in association with each other, for example, by storing association information that associates the internal data and the specific information.

< JPEG File >

Fig. 2 is a diagram illustrating an example of a format of a Joint Photographic Experts Group (JPEG) file conforming to JPEG.

The JPEG file is configured to store, for example, Exif metadata, a thumbnail image, extensible metadata platform (XMP) (registered trademark) metadata, MPF representing the storage positions (places) of the main image and the simplified display image, and the like, the main image, and the simplified display image. As a simple display image, for example, a screen thumbnail image may be employed.

< ISO base media File Format >

Fig. 3 is a diagram illustrating an example of an ISO base media file format.

The HEIF (ISO/IEC 23008-12) is a file format compliant with the ISO base media file format (ISO/IEC 14496-12), and thus the HEIF file is compliant with the ISO base media file format.

The ISO base media file format includes units called boxes as containers for storing data, and has a structure called a box structure.

The box includes a type (box type), actual data (data), and the like. The type indicates the type of actual data within the cartridge. As actual data, reproducible media data such as an image (still image, moving image), audio, or a subtitle (subtitle), an attribute name (field name), an attribute value (field value) of the attribute name (variable represented), and other various data may be employed.

Also, a box may be employed as the actual data. That is, the box may have a box as actual data, thus realizing a hierarchical structure.

The base media file conforming to the ISO base media file format may include an ftyp box, a moov box (MovieBox), a meta box (MetaBox), an mdat box (MediaDataBox), and the like. In the ftyp box, identification information for identifying a file format is stored. The moov box may store a trak box, etc. The meta box can store iinf box, iprp box, iref box, iloc box, etc. The mdat box can store media data (AV data) and other arbitrary data.

HEIF conforms to the ISO base media file format described above.

< HEIF document >

Fig. 4 is a diagram illustrating an example of a format of a HEIF file conforming to HEIF.

The HEIF file is roughly divided into an image item form and an image sequence form. Also, as the image item form, there are a single image form having only one item described later and an image collection form having a plurality of items.

The HEIF file in the form of an image item includes an ftyp box, a meta box, and an mdat box.

The HEIF file in the form of an image sequence includes ftyp box, moov box, and mdat box.

Note that the HEIF file may include not only one of a meta box and a moov box but also both of the meta box and the moov box.

The ftyp box stores identification information for identifying a file format indicating that the file is, for example, a HEIF file in the form of an image item or an image sequence.

In the meta box and moov box, metadata (e.g., metadata such as a storage location of media data) necessary for reproduction, management, and the like of the media data stored in the mdat box is stored.

Media data (AV data) and the like are stored in the mdat box.

In the digital camera 10, which of the HEIF files in the form of image items or the HEIF files in the form of image sequences is generated can be selected in accordance with the operation of the user, for example. Further, in the case where an image is encoded and stored in mdat of the HEIF file, only intra-coding is allowed in the image item form, and intra-coding and inter-coding are allowed in the image sequence form. Thus, for example, in the case of giving priority to high-speed access to data stored in the HEIF file, the HEIF file in the form of a generated image item may be selected, and in the case of giving priority to reduction in size (data amount) of the HEIF file, the HEIF file in the form of a generated image sequence may be selected.

Fig. 5 is a diagram illustrating an example of a format of a HEIF file in the form of an image item.

In the HEIF file in the form of an image item, information (for example, mif1 or the like) indicating that the HEIF file is in the form of an image item is stored (as an attribute value) in an ftyp box.

In the meta cartridge, an iinf cartridge, an iref cartridge, an iprp cartridge, and an iloc cartridge are stored.

In the iinf box, the number (attribute name and attribute value) of items as media data (AV data) stored in the mdat box, and the like are stored (indicated). An item is data stored in an mdat box of an HEIF file in the form of an image item, for example, a (picture) image is an item. In this specification, a single image is also referred to as a frame regardless of whether the image is a still image or a moving image. One frame is one item.

In the iref box, information indicating a relationship between items is stored. For example, in the mdat box, each of the corresponding main image, the corresponding screen thumbnail image, and the corresponding thumbnail image may be stored as an item. In a case where the item I1 as the main image, the item I2 as the screen thumbnail image, and the item I3 as the thumbnail image are stored in the mdat box, information indicating that the item I2 is the screen thumbnail image of the main image as the item I1 and information indicating that the item I3 is the thumbnail image of the main image as the item I1 are stored in the iref box.

In the iprp box, information on the characteristics of the items is stored.

In the iloc box, information about the storage location of the items stored in the mdat box is stored.

In the mdat box (of the HEIF file) in the form of an image item, for example, a frame of an image as an item is stored. One or more items may be stored in an mdat box. Further, frames as items may be encoded and stored in an mdat box. However, encoding of a frame as an item stored in an mdat box in the form of an image item is limited to intra-frame encoding only. As a coding scheme (codec) for coding a frame as an item, HEVC or the like can be adopted, for example.

Fig. 6 is a diagram illustrating an example of the iprp box in fig. 5.

In the iprp box, an ipco box and an ipma box regarding the characteristics of items are stored. In the ipco box, characteristics of items stored in the mdat box are stored, for example, codec information on a codec of an image as an item and image size information on a size of the image. In the ipma box, an index (pointer) of the items stored in the mdat box to the properties stored in the ipco box is stored.

Fig. 7 is a diagram illustrating an example of a format of a HEIF file in the form of an image sequence.

In the HEIF file in the form of an image sequence, information indicating that the HEIF file is in the form of an image sequence, such as msf1 or the like, is stored in an ftyp box.

In the moov box, a trak box is stored. In the trak box, information about tracks stored in the mdat box is stored.

A track comprises one single media data, such as an image or audio, reproduced according to the timeline. For example, a track includes one or more frames of images as an elementary stream. As for the tracks stored in the mdat box, a plurality of tracks, for example, tracks of images and audio recorded simultaneously, can be reproduced simultaneously.

The media data of a track comprises units called samples. A sample is the smallest unit (access unit) in the case of accessing media data in a HEIF file. Thus, media data in the HEIF file cannot be accessed in finer units than samples.

As for the media data of the image, for example, one frame or the like is one sample. Further, with respect to the audio media data, for example, one audio frame or the like defined in the standard of the audio media data is one sample.

In an mdat box (of a HEIF file) in the form of a sequence of images, the media data of a track is arranged in units called blocks. A block is a set of one or more samples arranged in logically contiguous addresses.

In the case where a plurality of tracks as media data are stored in an mdat box, the plurality of tracks are interleaved and arranged in units of blocks.

In the moov box, trak boxes that respectively manage tracks stored in the mdat box are stored.

In an mdat box in the form of a sequence of images, one or more tracks comprising media data such as images or audio are stored. In the mdat box, frames of images constituting a track may be encoded and stored. In encoding of frames constituting tracks stored in mdat boxes in the form of image sequences, a long group of pictures (GOP) may be employed as a GOP, and both intra-frame encoding and inter-frame encoding may be employed. As a codec for encoding frames constituting a track, for example, HEVC or the like can be employed.

Fig. 8 is a diagram illustrating an example of a trak box.

In the trak box, a tkhd box and an mdia box may be stored. In the tkhd box, header information of tracks managed by the trak box, such as the creation date and time of the track, is stored. In the mdia box, a minf box and the like are stored. In the minf cartridge, the stbl cartridge is stored. In the stbl box, stsd box, stsc box, stsz box, and stco box are stored, and these boxes store information for accessing samples of tracks, and thus blocks. In the stsd box, codec information of a codec with respect to the track is stored. In the stsc box, the block size (the number of samples of one block) is stored. In the stsz box, the sample size is stored. In the stco box, a block offset, that is, an offset of an arrangement place of each block of a track stored in the mdat box is stored.

Here, the HEIF file in the form of an image item is also referred to as an aggregate file, and the HEIF file in the form of an image sequence is also referred to as a sequence file. Also, the association-type HEIF file in the form of an image item is also referred to as an association-type set file, and the association-type HEIF file in the form of an image sequence is also referred to as an association-type sequence file.

In the digital camera 10, HEIF files (including an associated type HEIF file) storing the main image and either the required screen thumbnail image or the required thumbnail image or both the screen thumbnail image and the thumbnail image may be generated.

< Collection document >

Fig. 9 is a diagram illustrating an example of a general collection file storing main images and thumbnail images.

Here, the aggregate file generally refers to an aggregate file in which internal data in the aggregate file is not associated with specific information of external data.

Now, assume that a frame (entry) is encoded by HEVC and stored in the mdat box of the aggregate file.

In the ftyp box, as identification information for identifying a file format, heic indicating that the format is in the form of an image item and the codec is HEVC is stored.

In the iinf box, the number of items (number of items) stored in the mdat box is stored. In fig. 9, a total of four items (frames) -that is, the main image (hereinafter also referred to as main image Item #1) specified by Item ID #1, the main image Item #2, the thumbnail image (hereinafter also referred to as thumbnail image Item #101) specified by Item ID #101, and the thumbnail image Item #102 are stored in the mdat box. Thus, the number of terms is four. Note that the thumbnail image Item #101 is a thumbnail image of the main image Item #1, and the thumbnail image Item #102 is a thumbnail image of the main image Item # 2.

Also, in the iinf box, for example, an infe box is stored for each item stored in the mdat box. In the infe box, an item ID and an item type for specifying an item are registered. In fig. 9, there are the feed boxes of the main images Item #1 and Item #2 and the thumbnail images Item #101 and Item # 102.

In the iref box, for example, a thmb box is stored as information for associating items stored in the mdat box. In the thmb box, a reference source and a reference destination as information for associating the main image with the thumbnail image of the main image are stored in association with each other. In the thmb box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the thumbnail image of the main image specified by the item ID of the reference source. Thus, the item ID of the thumbnail image of the main image specified by the item ID represented by the reference source can be identified according to the reference destination corresponding to the reference source. Further, the item ID of the main image of the thumbnail image specified by the item ID represented by the reference destination can be identified according to the reference source corresponding to the reference destination.

As shown in fig. 6, the ipco box and the ipma box are stored in the iprp box. In the ipco box, as shown in fig. 6, frame characteristics, for example, codec information on a codec and image size information on a size, are stored as items stored in the mdat box. In the ipma box, as shown in fig. 6, an index of items stored in the mdat box to characteristics stored in the ipco box is stored.

In the iloc box, as shown in fig. 6, information on the storage location of the items in the mdat box is stored. In fig. 9, information indicating that the number of items is four is stored in the iloc box. Also, in the iloc box, the offset and size of the storage positions of the main images Item #1 and Item #2 and the thumbnail images Item #101 and Item #102 stored in the mdat box correspond to the Item ID and are stored.

Hereinafter, an associated-type aggregate file in which specific information of internal data and external data is stored in association with each other in the general aggregate file of fig. 9 will be described.

Fig. 10 is a diagram illustrating an example of a first association-type aggregate file.

Here, for example, it is assumed hereinafter that a RAW file storing a RAW image of a main image as internal data in the HEIF file is employed as external data to be associated with the main image.

In the first related-type aggregate file, related information is stored which associates the main image as internal data with specific information of a RAW file as external data (a RAW file storing a RAW image of the main image as internal data), and therefore, the specific information of the main image and the RAW file are stored in association with each other. Also, in the first association-type aggregate file, association information is stored in the meta box.

As the specific information of the RAW file as the external data, a file name of the RAW file, a universal unique identifier (uuid) issued to the RAW file, a Uniform Resource Locator (URL), and other arbitrary information that can specify the RAW file can be employed.

With respect to the first association-type aggregate file, the association information storage box storing the association information is defined as a new box to be stored in the meta box, and is stored in the meta box. In the association information storage box of the first association-type aggregate file, for example, association information for specifying the item ID of the main image corresponding to uuid as the specific information specifying the RAW file associated with the main image is stored. Also, in the associated information storage box, the number of main images (the number of main images) associated with the RAW file is stored. Since the number of main images stored in the associated information storage box is the number of main images associated with the RAW file, the number of main images is a value equal to or smaller than the number of main images stored in the mdat box.

In fig. 10, UUID of the RAW file of the main image Item #1 is UUID #1, and UUID of the RAW file of the main image Item #2 is UUID # 2. Now, assuming that the RAW file whose UUID is UUID # i is referred to as RAW file UUID # i, in fig. 10, the association information of the Item ID #1 of the main image Item #1 corresponding to UUID of the RAW file UUID #1 and the Item ID #2 of the main image Item #2 corresponding to UUID of the RAW file UUID #2 is stored in the association information storage box.

Fig. 11 is a diagram illustrating an example of a second association-type aggregate file.

In the second association-type aggregate file, similarly to the first association-type aggregate file, association information associating the main image as the internal data with the specific information of the RAW file as the external data is stored, and therefore the specific information of the main image and the RAW file are stored in association with each other. However, in the second association-type aggregate file, association information is stored in the mdat box.

With respect to the second association-type aggregate file, for example, association information similar to that of the first association-type aggregate file is stored as items in the mdat box. In fig. 11, the association information is stored in the mdat box as an item of item ID # 201.

As described above, in the second associative type aggregate file, since the association information as the Item #201 is stored in the meta box, the information stored in the mdat box is different from that of the normal aggregate file in fig. 9. In the second association-type collection file, metadata as association information of the Item #201 is stored in the meta box.

Specifically, in the second association-type aggregate file, the number of items stored in the iinf box and the iloc box is changed from four in the case of fig. 9 to five obtained by adding one to four, i.e., Item # 201. Also, an infe box for the Item #201 is added to the iinf box, and the offset and size of the storage location of the Item #201 are added to the iloc box. In the infe box for the Item #201, the Item ID #201 of the Item #201 is stored and Item type Identification Data Information (IDIF) indicating that the Item #201 is association information. The IDIF is a newly defined attribute value (field value) indicating that the item is association information.

Fig. 12 is a diagram illustrating an example of a third association-type aggregate file.

In the third association-type set file, for each piece of specific information, specific information of a RAW file as external data is stored as an item in an mdat box, and association information associating a main image as internal data with specific information of a RAW file as external data is stored in a meta box, so the specific information of the main image and the RAW file are stored in association with each other. However, in the third associated aggregate file, the associated information is information in which the item ID of the main image as an item corresponds to the item ID of the specific information (of the RAW file) as an item, and is stored in the cdsc box stored in the iref box in the meta box.

In the cdsc box, a reference source and a reference destination as information for associating an item as a main image with specific information of a RAW file of the main image can be corresponded and stored. In the cdsc box, the reference source represents an item ID of the main image, and the reference destination represents an item ID of specific information of an item of the RAW file of the main image specified by the item ID of the reference source.

In fig. 12, UUID #1 (which is UUID of the specific information of the RAW file of the main image Item #1) is stored as Item #201 in mdat, and UUID #2 (which is UUID of the specific information of the RAW file of the main image Item # 2) is stored as Item #202 in mdat. Further, a cdsc box storing association information associating the Item ID #1 of the main image Item #1 and the Item ID #201 of the specific information UUID #1 as a reference source and a reference destination is stored in the iref box, and a cdsc box storing association information associating the Item ID #2 of the main image Item #2 and the Item ID #202 of the specific information UUID #2 as a reference source and a reference destination is stored in the iref box.

< sequence document >

Fig. 13 is a diagram illustrating an example of a normal sequence file in which a track of a main image and a track of a thumbnail image of the main image are stored.

Here, the sequence refers to a sequence file in which internal data is not associated with specific information of external data.

Now, assume that a frame is encoded by HEVC and stored in the mdat box of the sequence file.

In the ftyp box, the HEVC indicating that the format is an image sequence form and the codec is HEVC is stored as identification information for identifying a file format.

In the moov box, as shown in fig. 7, trak boxes that respectively manage tracks stored in the mdat box are stored. In fig. 13, a track of the main image (hereinafter also referred to as track #1) designated by track ID #1 and a track #2 of the thumbnail image of the main image of track #1 are stored in the mdat box. Therefore, in the moov box, a trak box that manages track #1 and a trak box that manages track #2 are stored. The (frame of the) nth thumbnail image (from the beginning) of the track #2 is the thumbnail image of the nth main image of the track # 1.

For example, the sequence file is useful in a case where a main image of a plurality of frames is recorded as one track and a thumbnail image of a plurality of frames is recorded as one track, and in a case where, for example, continuous shooting is performed, the frames are obtained by the continuous shooting performed by the digital camera 10.

In the tkhd box of the trak box that manages the main image track #1, a track ID #1 specifying the track #1, the image size of the main image constituting the track #1, rotation information indicating the orientation of the digital camera 10 at the time when the main image was captured, and the creation date and time of the track #1 are stored. In the tkhd box of the trak box that manages track #2 of the thumbnail image, track ID #2 specifying track #2 and the creation date and time of track #2 are stored.

In the trak box, a tref box may be stored in addition to the tkhd box and the mdia box shown in fig. 7. In the tref box, a track ID for specifying other tracks associated with a track managed by the trak box in which the tref box is stored, information indicating the contents of the track, and the like are stored. In fig. 13, a tref box is provided in a trak box that manages track # 2. Then, in the tref box, information indicating that the other track related to the track #2 is the track #1(track _ ID is 1) and the data constituting the track #2 is a thumbnail image (the track #2 is a track of the thumbnail image) (type is thmb) is stored.

In the mdia box of the trak box, an hdlr box can be stored in addition to the minf box shown in fig. 8. In the hdlr box, information indicating the data type constituting a track managed by the trak box in which the hdlr box is stored. Information (pict) indicating that the data constituting the track #1 is a picture (frame) is stored in an hdlr box stored in a track box (in an mdia box stored) managing the track #1 of the main image, and information indicating that the data constituting the track #2 is a picture is stored in an hdlr box stored in a track box managing the track #2 of the thumbnail image.

The minf cassette is shown in FIG. 8.

Hereinafter, an associated type sequence file storing specific information of internal data and external data in association with each other in the normal sequence file of fig. 13 will be described.

Fig. 14 is a diagram illustrating an example of an association-type sequence file.

In the associated type sequence file, track #3 of a (elementary) stream (Meta ES) of uuid, which is specific information of a RAW file of external data, is added to an mdat box, and a trak box managing track #3 is added to an moov box.

Here, track #1 is a time series of one frame or a plurality of frames of the main image arranged on the timeline, and track #3 is a time series of uuid of the RAW file of the respective frames of the main image arranged on the timeline.

The nth uuid of the track #3 (from the beginning) is specific information of the RAW file of the nth frame of the main image of the track # 1. Further, (data of) a plurality of tracks stored in the mdat box can be synchronously reproduced according to the time point information on one timeline. Therefore, by storing the track #1 of the main image and the track #3 of (the stream of) the uuid of the RAW file of each frame of the main image constituting the track #1 in the mdat box, the nth frame of the main image of the track #1 and the uuid of the RAW file of (the frame of) the main image are stored in association with each other. In this case, it can be said that the frame of the main image of the track #1 and the uuid of the RAW file of the (frame of the) main image can be associated with each other by the time point information on the time line.

Note that, it is understood that the nth uuid (from the beginning) of the track #3 is specific information of the RAW file of the nth frame of the track #1, and the (frame of the) main image constituting the track #1 and the uuid constituting the track #3 are associated with each other in the arrangement order in the track.

In the association-type sequence file, since track #3 of uuid of the RAW file is added to the mdat box, a trak box that manages track #3 is added to the moov box.

In the trak box of track #3 of uuid which manages the RAW file, a tkhd box, a tref box, an mdia box, and the like are stored.

In the tkhd box of the trak box that manages track #3, track ID #3 specifying track #3 and the creation date and time of track #3 are stored.

In the tref box of the trak box managing the track #3, a track ID specifying another track related to the track #3 managed by the trak box storing the tref box, information indicating the content of the track #3, and the like are stored. Since uuid configuring the track #3 is specific information of the RAW file configuring the main image of the track #1 and the track #3 is related to the track #1, information indicating a track in which the other track related to the track #3 is the track #1(track _ ID 1) and the track #3 is metadata (here, specific information) (type cdsc) is stored in the tref box managing the track #3 of fig. 14.

In the mdia box of the trak box managing track #3, an hdlr box and a minf box are stored. In the trak box that manages the track #3, information (meta) indicating that the data constituting the track #3 is metadata (of the main image) is stored in the hdlr box, and the stsc box, stsz box, and stco box with respect to the track #3 are stored in the minf box.

< creation and reproduction of HEIF File >

Fig. 15 is a flowchart illustrating an outline of an example of the generation processing of generating the association-type HEIF file.

In the generation processing, in step S11, the file control unit 43 generates uuid of the specific information of the RAW file as the frame of the main image, and the processing proceeds to step S12.

In step S12, the file control unit 43 assigns uuid generated in step S11 to the RAW file of the frame of the main image, and the processing proceeds to step S13.

In step S13, the file control unit 43 generates an association-type HEIF file in which the frames of the main image and uuid of the RAW file of the frames are stored in association with each other in the HEIF file, and terminates the generation processing.

Fig. 16 is a flowchart illustrating an outline of an example of reproduction processing of reproducing the associated type HEIF file.

In the reproduction processing, in step S21, the file control unit 43 generates a handle list of handles (handles) for identifying the frames of the main image in the HEIF file stored in the medium 14, respectively, for example, and the processing proceeds to step S22.

Here, the handle of the frame of the main image includes the file name of the HEIF file in which the frame is stored. The handle to the frame (item) of the main image stored in the collection file further includes an item ID of the frame. The handle to the frame of the main image stored in the sequence file includes time point information of the frame. With the handle of the frame of the host, the frame for the handle can be uniquely identified (specified).

Note that the handle of the frame of the main image stored in the sequence file may include the track ID of the track containing the frame and the order of the frames in the track (where the frame is arranged), instead of the time point information of the frame.

Whether one track or a plurality of tracks of frames including the main image are stored in the sequence file, the time point information of each frame is unique. Therefore, according to the point-in-time information of the frames, even if a plurality of tracks are stored in the sequence file, the frames of the point-in-time information included in the handle can be uniquely specified from the frames constituting the plurality of corresponding tracks. Therefore, in the case where the time point information of the frame of the main image is included in the handle of the frame, even if there is no track ID of the track where the frame exists, the frame corresponding to the time point information can be uniquely specified.

The handle list may be generated for all frames of the main image stored in the HEIF file saved in the medium 14, or may be generated only for frames limited under certain conditions, such as frames having a certain creation date and time.

After generating the handle list, the file control unit 43 refers to the handle list as necessary to access the HEIF file.

In step S22, for example, the UI control unit 47 waits for the user to, for example, operate the digital camera 10 to display a thumbnail image, and requests the file control unit 43 to display the thumbnail image. The file control unit 43 reads (a frame of) the thumbnail image of the frame of the main image identified by the handle of the handle list from the HEIF file in response to a display request of the thumbnail image from the UI control unit 47. Then, the file control unit 43 displays a list of thumbnail images read from the HEIF file on, for example, the liquid crystal panel 19 (fig. 1), and the process proceeds from step S22 to step S23.

In step S23, for example, after waiting for the user to select (a frame of) a desired thumbnail from the list of thumbnail images, the UI control unit 47 requests the main image corresponding to the thumbnail image selected by the user from the file control unit 43. In response to a request for the main image from the UI control unit 47, the file control unit 43 reads the main image from the HEIF file. The file control unit 43 can display the main image read from the HEIF file on the liquid crystal panel 19 as needed.

Alternatively, the UI control unit 47 requests the file control unit 43 to supply uuid of the RAW file of the main image corresponding to the thumbnail image selected by the user. In response to a request for uuid from the UI control unit 47, the file control unit 43 reads uuid from the association-type HEIF file. The file control unit 43 can access the RAW file specified by the uuid read from the associated type HEIF file as needed.

Fig. 17 is a flowchart illustrating an example of reproduction processing of reproducing an aggregate file.

In step S31, the file control unit 43 acquires an item ID (hereinafter also referred to as a reproduction target item ID) of a reproduction target image as an image (item) to be reproduced, and the processing proceeds to step S32.

In acquiring the reproduction target item ID, for example, a main image identified by an arbitrary handle of the handle list, a thumbnail image of the main image, a thumbnail image selected by the user from the list of thumbnail images (hereinafter also referred to as a selected thumbnail image), a main image of the selected thumbnail image, or the like is set as the reproduction target image, and an item ID of the reproduction target image (reproduction target item ID) is acquired.

In step S32, the file control unit 43 reads the reproduction target image according to the reproduction target item ID acquired in step S31.

When reading the reproduction target images, the reproduction target images specified by the reproduction target item ID are read from the aggregate file.

Fig. 18 is a flowchart illustrating an example of the process of reading the reproduction target image in step S32 of fig. 17.

In step S41, the file control unit 43 searches the iloc box of the aggregate file (fig. 9 to 12) for the reproduction target item ID, and the processing proceeds to step S42.

In step S42, in the iloc box, the file control unit 43 reads the offset and size corresponding to the reproduction target item ID searched for in step S41, and the process proceeds to step S43.

In step S43, the file control unit 43 reads the reproduction target image stored in the mdat box of the aggregate file in accordance with the offset and size corresponding to the reproduction target item ID, and the process terminates.

Fig. 19 is a flowchart illustrating a first example of the process of acquiring a reproduction target item ID in step S31 of fig. 17.

That is, fig. 19 illustrates an example of setting a thumbnail image as a reproduction target image and acquiring an item ID of the thumbnail image as the reproduction target image.

Note that, in fig. 19, it is assumed that, for example, the file control unit 43 recognizes the item ID of the main image of the thumbnail image from the sentence handle as the reproduction target image.

In step S51, the file control unit 43 searches for a thmb box whose reference source matches the item ID of the main image among thmb boxes in iref boxes of the aggregate file (fig. 9 to 12), and the process proceeds to step S52.

In step S52, the file control unit 43 reads the reference destination in the thmb box whose reference source matches the item ID of the main image (the thmb box has been searched for in step S51) as the item ID of the thumbnail image as the reproduction target image, and the process terminates.

Fig. 20 is a flowchart illustrating a second example of the process of acquiring a reproduction target item ID in step S31 of fig. 17.

That is, fig. 20 illustrates an example of setting the main image as the reproduction target image and acquiring the item ID of the main image as the reproduction target image.

Note that, in fig. 20, for example, it is assumed that the user selects a thumbnail image (selected thumbnail image) from the list of thumbnail images, and the file control unit 43 identifies the item ID of the selected thumbnail image.

In step S61, the file control unit 43 searches for a thmb box whose reference destination matches the item ID of the selected thumbnail image among thmb boxes in iref boxes of the aggregate file (fig. 9 to 12), and the process proceeds to step S62.

In step S62, the file control unit 43 reads the reference source of the thmb box whose reference destination matches the item ID of the selected thumbnail image (the thmb box has been searched for in step S61) as the item ID of the main image as the reproduction target image, and the process terminates.

Fig. 21 is a flowchart illustrating an example of processing of acquiring the uuid of the specific information of the RAW file as the predetermined main image from the first association-type set file of fig. 10.

Note that in fig. 21, for example, it is assumed that the file control unit 43 recognizes the item ID of the predetermined main image by using a handle list or the like.

In step S71, the file control unit 43 searches the associated information in the associated information storage box of the first associated-type aggregate file (fig. 10) for the item ID of the predetermined main image, and the processing proceeds to step S72.

In step S72, the file control unit 43 reads the uuid corresponding to the item ID of the predetermined main image searched for in step S71 in the association information, and the processing is terminated.

The file control unit 43 can access the RAW file of the predetermined main image by using the uuid read as described above.

Fig. 22 is a flowchart illustrating an example of processing of acquiring the uuid of the specific information of the RAW file as the predetermined main image from the second related-type set file of fig. 11.

Note that in fig. 22, for example, it is assumed that the file control unit 43 recognizes the item ID of the predetermined main image by using the handle list or the like.

In step S81, the file control unit 43 searches for an nfet box having an item type IDIF indicating that the item is associated information among the ife boxes of the second association-type aggregate file (fig. 11), and the process proceeds to step S82.

In step S82, the file control unit 43 reads the item ID as the association information of the item from the infe box with the item type IDIF searched for in step S81, and the processing proceeds to step S83.

In step S83, the file control unit 43 searches the iloc box of the second association-type aggregate file for the item ID of the association information read in step S82, and the processing proceeds to step S84.

In step S84, in the iloc box, the file control unit 43 reads the offset and the size corresponding to the item ID of the associated information searched for in step S83, and the process proceeds to step S85.

In step S85, the file control unit 43 reads the association information as items stored in the mdat box of the second association-type aggregate file in accordance with the offset and size corresponding to the item ID of the association information, which have been read in step S84, and the process proceeds to step S86.

In step S86, the file control unit 43 searches the association information read in step S85 for the item ID of the predetermined main image, and the processing proceeds to step S87.

In step S87, the file control unit 43 reads the uuid corresponding to the item ID of the predetermined main image searched for in step S86 in the association information, and the processing is terminated.

The file control unit 43 can access the RAW file of the predetermined main image by using the uuid read as described above.

Fig. 23 is a flowchart illustrating an example of processing of acquiring the uuid of the specific information of the RAW file as the predetermined main image from the third related-type set file of fig. 12.

Note that in fig. 23, for example, it is assumed that the file control unit 43 recognizes the item ID of the predetermined main image by using the handle list or the like.

In step S91, the file control unit 43 searches the cdsc box in the iref boxes of the third associated-type aggregate file (fig. 12) for a cdsc box whose reference source matches the item ID of the predetermined main image, and the process proceeds to step S92.

In step S92, the file control unit 43 reads the reference destination in the cdsc box (which has been searched in step S91) matching the reference source with the item ID of the predetermined main image as the item ID of the specific information of the RAW file of the predetermined main image as an item, and the processing proceeds to step S93.

In step S93, the file control unit 43 searches the iloc box of the third associated aggregate file for an item ID as the specific information of the item, which has been read in step S92, and the process proceeds to step S94.

In step S94, in the iloc box, the file control unit 43 reads the offset and size corresponding to the item ID of the specific information, which has been searched for in step S93, and the process proceeds to step S95.

In step S95, the file control unit 43 reads the uuid of the specific information of the RAW file of the predetermined main image stored in the mdat box of the third association-type set file in accordance with the offset and the size corresponding to the item ID of the specific information, which have been read in step S94, and the process is terminated.

The file control unit 43 can access the RAW file of the predetermined main image by using the uuid read as described above.

Fig. 24 is a flowchart illustrating an example of a process of acquiring a list of item IDs of a main image from a collection file.

For example, in the case of generating a handle list, processing of acquiring a list of item IDs of the main image from the collection file is performed.

In step S101, the file control unit 43 reads the item IDs from all of the infe boxes of the iinf boxes of the aggregate file (fig. 9 to 12), registers the item IDs in a list of item IDs of the main image (hereinafter also referred to as a main image list), and the process proceeds to step S102.

In step S102, the file control unit 43 reads the item ID as the reference destination from all the boxes in the iref box of the aggregate file, excludes the item ID from the main image list, and the process is terminated.

After the above processing, the item ID registered in the main image list becomes the item ID of the main image.

Fig. 25 is a flowchart illustrating an example of processing of reproducing a thumbnail image of (a frame of) a main image for predetermined point-in-time information from a sequence file.

Note that, in fig. 25, for example, it is assumed that the file control unit 43 recognizes time point information (or order) of a predetermined main image by using a handle list or the like.

In step S111, the file control unit 43 searches for a trak box in the moov box of the sequence file (fig. 13 and 14) in which information indicating that the data constituting the track is a thumbnail image is stored in the tref box, that is, a trak box of type thmb in the tref box as a trak box of a track managing a thumbnail image of a main image for predetermined point-in-time information, and the process proceeds to step S112.

In step S112, the file control unit 43 reads the track ID in the tkhd box in the trak box that has been searched in step S111 as the track ID of the track of the thumbnail image of the main image for the predetermined point-in-time information, and the processing proceeds to step S113.

In step S113, the file control unit 43 reproduces the track having the track ID read in step S112, and acquires (the frame of) the thumbnail image for the predetermined point-in-time information (or order) from the track as the thumbnail image of the main image for the predetermined point-in-time information, and the process is terminated.

Note that the process of reproducing the images stored in the sequence file is similar to the process of reproducing the moving image of the MP4 file.

Fig. 26 is a flowchart illustrating an example of processing of acquiring the uuid of the specific information of the RAW file as (the frame of) the predetermined main image from the associated-type sequence file.

Note that in fig. 26, for example, it is assumed that the file control unit 43 recognizes time point information (or order) of a predetermined main image by using a handle list or the like.

In step S121, the file control unit 43 searches the trak box in the moov box of the associated-type progressive file (fig. 14) for a trak box in which information indicating that the data constituting the track is the specific information is stored in the tref box, that is, a trak box of type cdsc in the tref box, as a trak box that manages the track of the specific information, and the process proceeds to step S122.

In step S122, the file control unit 43 reads the track ID in the tkhd box in the trak box that has been searched in step S121 as the track ID of the track of the specific information, and the processing proceeds to step S123.

In step S123, the file control unit 43 acquires uuid as specific information of the time point information (or order) for the predetermined main image from the track having the track ID read in step S122 as uuid of the RAW file of the predetermined main image, and the processing is terminated.

The file control unit 43 can access the RAW file of the predetermined main image by using the uuid acquired as described above.

As described above, the file control unit 43 generates and reproduces an associated type HEIF file in which the main image in the HEIF file conforming to HEIF and the specific information specifying the external data other than the HEIF file, which is to be associated with the main image, are stored in association with each other in the HEIF file. Accordingly, the main image stored in the HEIF file can be associated with external data other than the HEIF file.

Further, in the case of using uuid as the specific information, even if the file name of the external data is changed, the association between the main image in the HEIF file and the external data whose file name has been changed can be maintained by uuid.

< storage of specific information assigned to external data >

Fig. 27 is a diagram illustrating an example of storing uuid in a RAW file in the case of taking the RAW file of the main image as external data and generating an association-type set file.

Note that, in fig. 27, the first association-type aggregate file is adopted as the association-type aggregate file.

The RAW file has an area called a marker note (MakerNote) as a partial area of an area for storing additional information of Exif as metadata.

The file control unit 43 may store uuid allocated to the RAW file in, for example, MakerNote of the RAW file.

In fig. 27, the main image Item #1, Item #2, Item #3, and Item #4 as four items are stored in the associated type set file, and the RAW files #1, #2, #3, and #4 of the RAW image in which the main image Item #1, Item #2, Item #3, and Item #4 are stored, respectively, are generated. Then, the UUID # i is assigned to the RAW file # i, and as association information associating the main image Item # i with the UUID # i of the RAW file # i of the main image Item # i, association information specifying that the Item ID # i of the main image Item # i is associated with the UUID # i of the RAW file # i associated with the main image Item # i is stored in the association information storage box.

Fig. 28 is a diagram illustrating an example of storing uuid in a RAW file in the case of taking the RAW file of the main image as external data and generating an association-type sequence file.

The file control unit 43 may also store uuid allocated to the RAW file in MakerNote of the RAW file in the case of generating the association-type sequence file, similarly to the case of generating the association-type set file shown in fig. 27.

In fig. 28, a track #1 including main images #1, #2, #3, and #4 as four frames is stored in an associated type sequence file, and RAW files #1, #2, #3, and #4 storing RAW images of the main images #1, #2, #3, and #4, respectively, are generated. Then, the UUID # i is assigned to the RAW file # i, and the track #3 arranged and configured such that the UUID # i of the RAW file # i has the same time point information as that of the main image # i corresponding to the RAW file # i is stored in the association type sequence file.

As described above, the track #3 is configured such that the UUID # i of the RAW file # i is arranged to have the same time point information as that of the main image # i corresponding to the RAW file # i, so that the i-th main image # i of the track #1 and the i-th UUID # i of the track #3 (i.e., the UUID # i of the RAW file # i of the main image # i) are associated with each other and stored in the association-type sequence file.

In the above description, the RAW file of the main image is employed as the external data; however, other data may be employed as the external data. As the external data, for example, a file or the like storing audio (sound) recorded together with imaging of the main image may be employed. As the file storing the audio, for example, a WAV file in a WAV format, an MP4 file in an MP4 format, or the like can be adopted. Hereinafter, it is assumed that, for example, a WAV file is employed as a file storing audio.

Fig. 29 is a diagram illustrating an example of storing uuid in a WAV file in the case of taking the WAV file of the main image (the WAV file storing audio recorded with imaging of the main image) as external data and generating an association-type set file.

Note that, in fig. 29, the first association-type aggregate file is adopted as the association-type aggregate file.

The WAV file has an area called a List chunk (List chunk) as a partial area of an area describing metadata.

The file control unit 43 may store uuid assigned to the WAV file in, for example, a list block of the WAV file.

In fig. 29, the main images Item #1, Item #2, Item #3, and Item #4 as four items are stored in the associated type set file, and the WAV files #1, #2, #3, and #4 of the main images Item #1, Item #2, Item #3, and Item #4 are generated. Then, the UUID # i is assigned to the WAV file # i, and as association information associating the main image Item # i with the UUID # i of the WAV file # i of the main image Item # i, association information specifying that the Item ID # i of the main image Item # i is associated with the UUID # i of the WAV file # i associated with the main image Item # i is stored in the association information storage box.

Fig. 30 is a diagram illustrating an example of storing uuid in a WAV file in the case of taking the WAV file of the main image as external data and generating an association-type sequence file.

Even in the case of generating an association-type sequence file, the file control unit 43 can store uuid assigned to the WAV file in the list block of the WAV file, similarly to the case of generating an association-type aggregate file shown in fig. 29.

In fig. 30, track #1 including main images #1, #2, #3, and #4 as four frames is stored in an associated type sequence file, and WAV files #1, #2, #3, and #4 of the main images #1, #2, #3, and #4 are generated. Then, UUID # i is assigned to WAV file # i, and track #3 arranged and configured such that UUID # i of RAW file # i has the same time point information as that of the main image # i corresponding to RAW file # i is stored in the association type sequence file.

As described above, the track #3 is configured such that the UUID # i of the RAW file # i is arranged to have the same time point information as that of the main image # i corresponding to the RAW file # i, so that the i-th main image # i of the track #1 and the i-th UUID # i of the track #3 (i.e., the UUID # i of the WAV file # i of the main image # i) are associated with each other and stored in the association-type sequence file.

Note that the present technology can be applied to, for example, ISO base media files having a box structure other than the HEIF file, MP4 files, Miaf files, and the like, in addition to the HEIF file.

Further, the present technology can be applied to, for example, a file in which an image (main image) and other images of which image resolutions are reduced are stored, a file without a box structure, or the like.

Also, the present technology can be applied to a case where external data is associated with a screen thumbnail image or a thumbnail image in the HEIF file, in addition to a case where the external data is associated with a main image in the HEIF file.

Further, the present technology can also be applied to, for example, a case where external data is associated with internal data other than an image such as a main image in a HEIF file.

< description of computer to which the present technology is applied >

Next, a series of processes of each block constituting the signal processing unit 13 (fig. 1), such as the above-described file control unit 43, may be performed by hardware or may be performed by software. In the case where a series of processes is executed by software, a program configuring the software is installed on a computer or the like.

Fig. 31 is a block diagram illustrating a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

The program may be recorded in advance in the hard disk 905 or the ROM 903 as a recording medium built in the computer.

Alternatively, the program may be stored (recorded) in a removable recording medium 911 driven by a drive 909. Such a removable recording medium 911 may be provided as so-called packaged software. Here, examples of the removable recording medium 911 include a floppy disk, a compact disc read only memory (CD-ROM), a magneto-optical (MO) disk, a Digital Versatile Disc (DVD), a magnetic disk, a semiconductor memory, and the like.

Note that the program may be installed on the computer from the removable recording medium 911 as described above, and further, may be downloaded to the computer via a communication network or a broadcast network and installed on the built-in hard disk 905. That is, for example, the program may be wirelessly transmitted from a download site to the computer via an artificial satellite for digital satellite broadcasting, or may be transmitted to the computer via a network such as a Local Area Network (LAN) or the internet by cable.

The computer contains a Central Processing Unit (CPU)902, and an input/output interface 910 is connected to the CPU 902 via a bus 901.

When a command is input by a user via the input/output interface 910 according to an operation of the input unit 907 or the like, the CPU 902 executes a program stored in the Read Only Memory (ROM)903 according to the command. Alternatively, the CPU 902 loads a program stored in a hard disk 905 into a Random Access Memory (RAM)904 and executes the program.

Accordingly, the CPU 902 executes the processing according to the above-described flowchart or the processing executed by the configuration of the above-described block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 or transmits the processing result from the communication unit 908 via the input/output interface 910 as necessary, or further, records the processing result in the hard disk 905, for example.

Note that the input unit 907 includes a keyboard, a mouse, a microphone, and the like. Further, the output unit 906 includes a Liquid Crystal Display (LCD), a speaker, and the like.

Here, in this specification, the processing executed by the computer according to the program does not necessarily need to be executed in time series in the order described in the flowcharts. That is, the processing executed by the computer according to the program also includes processing executed in parallel or individually (for example, parallel processing or processing by an object).

Further, the program may be processed by one computer (processor) or may be subjected to distributed processing by a plurality of computers. Also, the program may be transferred to a remote computer and executed.

In this specification, a system refers to a collection of a plurality of components (devices, modules (components), and the like), and it does not matter whether or not all the components are in the same housing. Therefore, each of a plurality of devices accommodated in a single housing and connected via a network, and one device in which a plurality of modules are accommodated in one housing are systems.

Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

For example, the present technology may employ a configuration of cloud computing in which one function is shared and collectively handled by a plurality of devices via a network.

Further, each step described in the above-described flowcharts may be executed by one device, or may be shared and executed by a plurality of devices.

Also, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be shared and executed by a plurality of devices in addition to being executed by one device.

Further, the effects described in this specification are merely examples and are not limiting, and other effects may be provided.

Note that the present technology can also be configured as follows.

<1>

A file generating apparatus includes

A file control unit that generates an associated type high-efficiency image file format (HEIF) file in which specific information of an image conforming to the HEIF in the HEIF file and external data other than a specified HEIF file are stored in association with each other, the external data being associated with the image.

<2>

The file generating device according to <1>, wherein the file controlling unit generates an association type HEIF file in which association information associating an image with specific information of external data is stored.

<3>

The file generating device according to <2>, wherein the file controlling unit stores association information in which an item ID specifying an image corresponds to the specific information in the association-type HEIF file.

<4>

The file generating apparatus according to <3>, wherein the file controlling unit stores the association information in a meta box or an mdat box of the association type HEIF file.

<5>

The file generating apparatus according to <2>, wherein the file control unit

Storing specific information in mdat box of the associative type HEIF file, an

The association information in which the item ID specifying the image and the item ID specifying the specific information stored in the mdat box are associated is stored in the meta box of the association-type HEIF file.

<6>

The file generating device according to <1>, wherein the file control unit stores a track of specific information of the external data associated with each frame constituting a track of the image in an mdat box of the association type HEIF file.

<7>

The file generating device according to <6>, wherein each frame constituting a track of the image and the specific information constituting a track of the specific information are associated with each other by time point information on the time line or an order of arrangement in the track.

<8>

The file generating apparatus according to any one of <1> to <7>, wherein the image and the other image based on the image are stored in an associated type HEIF file.

<9>

A file generation method comprises

An associated type high-efficiency image file format (HEIF) file is generated in which specific information of an image conforming to HEIF in the HEIF file and external data other than the specified HEIF file are stored in association with each other, the external data being associated with the image.

<10>

A program for causing a computer to function as follows

A file control unit that generates an associated type high-efficiency image file format (HEIF) file in which specific information of an image conforming to the HEIF in the HEIF file and external data other than a specified HEIF file are stored in association with each other, the external data being associated with the image.

<11>

A file reproducing apparatus includes

And a file control unit reproducing an associated type high-efficiency image file format (HEIF) file in which specific information of an image conforming to HEIF in the HEIF file and external data other than the specified HEIF file are stored in association with each other, the external data being associated with the image.

<12>

The file reproducing apparatus according to <11>, wherein the file control unit reads the specific information of the external data associated with the predetermined image from the association type HEIF file, and the association information data associating the image with the specific information of the external data is stored in the association type HEIF file.

<13>

The file reproducing apparatus according to <12>, wherein association information associating the item ID of the specified image with the specific information is stored in an association-type HEIF file, and

the file control unit reads specific information corresponding to an item ID of a predetermined image in the association information.

<14>

The file reproducing apparatus according to <13>,

wherein the associated information is stored in a meta box or an mdat box of the associated type HEIF file, an

The file control unit reads specific information corresponding to an item ID of a predetermined image in the association information from the meta box or the mdat box.

<15>

The file reproducing apparatus according to <12>,

wherein specific information is stored in the mdat box of the associative type HEIF file, an

Association information that associates the item ID of the specified image with the item ID of the specified information stored in the mdat box is stored in the meta box of the association-type HEIF file, an

The file control unit reads specific information specified by an item ID of the specific information corresponding to an item ID of a predetermined image among the associated information stored in the meta box from the mdat box.

<16>

The file reproducing apparatus according to <11>,

wherein a track of specific information of external data associated with each frame constituting the track of an image is stored in an mdat box of an associated type HEIF file, and

each frame constituting a track of an image and specific information constituting a track of the specific information are associated with each other by time point information on a time line or an order of arrangement in the track, an

The file control unit acquires, from the track of the specific information, time point information on a timeline of the predetermined frame or specific information of an order of arrangement of the predetermined frame in the track.

<17>

The file reproducing apparatus according to any one of <11> to <16>, wherein the image and the other image based on the image are stored in an associated type HEIF file.

<18>

A file reproducing method includes

An associated type high-efficiency image file format (HEIF) file is reproduced in which an image complying with HEIF in the HEIF file and specific information specifying external data other than the HEIF file, the external data being associated with the image, are stored in association with each other.

<19>

A program for causing a computer to function as follows

And a file control unit reproducing an associated type high-efficiency image file format (HEIF) file in which specific information of an image conforming to HEIF in the HEIF file and external data other than the specified HEIF file are stored in association with each other, the external data being associated with the image.

List of reference numerals

10 digital camera

11 optical system

13 Signal processing unit

14 medium

15, 16 interface

17 push button/key

18 touch panel

19 liquid crystal panel

20 viewfinder

21 interface

41 optical system/image sensor control unit

42 code control unit

43 document control unit

44 media control unit

45 operation control unit

46 display control unit

47 UI control Unit

901 bus

902 CPU

903 ROM

904 RAM

905 hard disk

906 output unit

907 input unit

908 communication unit

909 driver

910 input/output interface

911 removable recording medium