阅读说明：本技术 短轴心脏mr电影回放切片堆叠的切片对准 (Slice alignment for short axis cardiac MR cine playback slice stacking ) 是由 J·彼得斯 R·J·威斯 T·维塞尔 F·M·韦伯 于 2020-03-24 设计创作，主要内容包括：针对短轴心脏磁共振电影回放切片堆叠描述了切片对准方法,其不需要诸如长轴扫描或全3D扫描的额外扫描,并且能够处理具有复杂形状的心脏结构。这两种方法不需要遵循二次曲率函数的轮廓,并且非常适于使用可变形表面模型获得心脏结构的分割。亦即,这种可变形的表面模型不能,但也不期望,完全适应于由于切片未对准而造成的心脏结构的边界中的“Z字形”模式。在移除或减小图像切片之间的未对准后,这种可变形表面模型可以更好地适应于图像数据中的心脏结构,并且从而提供心脏结构的更好的分割。(Slice alignment methods are described for short axis cardiac magnetic resonance cine loop slice stacking that do not require additional scans such as long axis scans or full 3D scans and are capable of processing cardiac structures having complex shapes. Both methods do not need to follow the contour of a quadratic curvature function and are well suited for obtaining a segmentation of cardiac structures using deformable surface models. That is, such deformable surface models cannot, but are also not expected to, fully adapt to "zig-zag" patterns in the boundaries of cardiac structures due to slice misalignment. Such a deformable surface model may better adapt to cardiac structures in the image data after removing or reducing misalignments between the image slices and thereby provide a better segmentation of the cardiac structures.)

1. A system (100) for slice alignment of a short axis cardiac magnetic resonance cine playback slice stack, comprising:

an input interface (120) for accessing image data (030) of a set of input image slices acquired using a short-axis cardiac magnetic resonance cine-playback protocol;

a processor subsystem (140) configured to:

-accessing training model data (050) defining a machine training model, wherein the machine training model is trained on training data comprising image data of a set of training image slices acquired using a short-axis cardiac magnetic resonance cine-playback protocol, wherein one or more adjacent image slices are mutually misaligned, and wherein the training data further comprises shift values for reducing the mutual misalignment by shifting one or more of the image slices;

-applying the machine training model to a set of neighboring image slices of the input image slice set (400), thereby obtaining at least one shift value for at least one of the image slices of the set of neighboring image slices; and is

-shifting the image slice based on the shift value.

2. The system (100) according to claim 1, wherein the processor subsystem (140) is configured to:

-applying the machine training model to the set of adjacent image slices (400) in the input image slice set to obtain a series of shift values;

-removing an offset or linear trend from the series of shift values;

-shifting respective image slices of the set of neighboring image slices based on respective shift values of the series of shift values.

3. The system (100) according to claim 1 or 2, wherein the machine training model is configured and trained for use as further input position information indicating a position of the respective set of neighboring image slices relative to a cardiac structure shown in the input image slice set.

4. The system (100) according to any one of claims 1 to 3, wherein the machine training model is configured and trained to serve as further input angle information indicative of an orientation of a cardiac structure shown in the input image slice set with respect to a coordinate system associated with the input image slice set.

5. The system (100) according to claim 3 or 4, wherein the processor subsystem (140) is configured to obtain at least one of the position information and the angle information by segmenting the cardiac structure in the input image slice set, for example by applying a deformable surface model to the image data to segment the cardiac structure in the input image slice set.

6. The system (100) according to claim 5, wherein the processor subsystem (140) is configured to mask portions of the image data of the input image slice set that do not belong to the cardiac structure prior to applying the machine training model to the set of adjacent image slices of the input image slice set.

7. The system (100) according to any one of claims 1 to 6, wherein the input image slice set is a first image slice set (400), wherein the input interface is configured for accessing image data of a second image slice set (402) acquired during a different cardiac phase than the first image slice set, and wherein the machine training model is configured and trained to use spatially corresponding samples of the first and second image slice sets as joint input.

8. A computer-readable medium (800) comprising transient or non-transient data (810) defining a machine training model, wherein the machine training model is configured and trained to be applied to a set of adjacent image slices in a set of image slices acquired using a short-axis cardiac magnetic resonance cine-loop protocol, wherein the machine training model is trained to output shift values for reducing mutual misalignment if the set of adjacent image slices are misaligned with each other.

9. A system (100) for slice alignment of a short axis cardiac magnetic resonance cine playback slice stack, comprising:

an input interface (120) for accessing image data (030) representing a set of image slices acquired using a short-axis cardiac magnetic resonance cine playback protocol;

a processor subsystem (140) configured to:

-accessing surface model data (060) defining a deformable surface model for segmenting cardiac structures in a short axis cardiac MR cine playback slice stack, wherein a deformability of the surface model is constrained by shape normalization;

-adapting the surface model to the cardiac structure by detecting boundary points of the cardiac structure in the image data and deforming the surface model towards the boundary points to obtain an adapted surface model adapted in shape to the cardiac structure in the image data; and is

-shifting (S1-S3) at least one image slice relative to other image slices, such that the boundary points in the image slices obtain an improved match to the cross-sectional representation of the surface model in the respective image slice.

10. The system (100) according to claim 9, wherein the processor subsystem (140) is configured to: deforming the surface model towards the boundary points of the cardiac structure based on a cost function penalizing distances of the surface model to the boundary points; and shifting the at least one image slice relative to the other image slices such that the matching is improved according to the cost function.

11. The system (100) according to claim 9 or 10, wherein the processor subsystem (140) is configured for iterative slice alignment by repeating the adaptation to the surface model and the shifting of the at least one image slice at least twice.

12. The system (100) according to any one of claims 9 to 11, wherein the processor subsystem (140) is configured to:

-after adapting the surface model, obtaining a series of shift values for respective image slices of the set of image slices to obtain the improved match with the cross-sectional representation of the surface model in the respective image slice;

-removing an offset or linear trend from the series of shift values;

-shifting the respective image slice based on the respective shift value of the series of shift values.

13. A computer-implemented method (600) for slice alignment of a short-axis cardiac magnetic resonance cine-loop slice stack, comprising:

-accessing (610) image data of an input image slice set acquired using a short-axis cardiac magnetic resonance cine-loop protocol;

-accessing (620) training model data defining a machine training model, wherein the machine training model is trained on training data comprising image data of a set of training image slices acquired using a short-axis cardiac magnetic resonance cine-playback protocol, wherein one or more neighboring image slices are mutually misaligned, and wherein the training data further comprises shift values for reducing the mutual misalignment by shifting one or more of the image slices;

-applying (630) the machine training model to a set of neighboring image slices of the input image slice set, thereby obtaining at least one shift value for at least one of the image slices of the set of neighboring image slices; and is

-shifting (640) the image slice based on the shift value.

14. A computer-implemented method (700) for slice alignment of a short-axis cardiac magnetic resonance cine-loop slice stack, comprising:

-accessing (710) image data representing a set of image slices acquired using a short-axis cardiac magnetic resonance cine-loop protocol;

-accessing (720) surface model data defining a deformable surface model for segmenting cardiac structures in a short axis cardiac MR cine playback slice stack, wherein a deformability of the surface model is constrained by a shape normalization;

-adapting (730) the surface model to the cardiac structure by detecting boundary points of the cardiac structure in the image data and deforming the surface model towards the boundary points to obtain an adapted surface model adapted in shape to the cardiac structure in the image data; and is

-shifting (740) at least one image slice relative to other image slices such that the boundary points in the image slices obtain an improved match to the cross-sectional representation of the surface model in the respective image slice.

15. A computer-readable medium (800) comprising transient or non-transient data (810) representing a computer program comprising instructions for causing a processor system to perform the method according to claim 13 or 14.

Technical Field

The invention relates to a system and computer-implemented method for slice alignment of a short-axis cardiac Magnetic Resonance (MR) cine playback slice stack. The invention also relates to a machine-trained model for slice alignment of a short-axis cardiac MR cine-slice stack, and to a computer-readable medium comprising instructions for performing one of the computer-implemented methods.

Background

Cardiac MR images are typically acquired in a series of cine-loop (cine) acquisitions of short-axis slices, and the time delay between multiple breaths to keep a single slice acquisition leads to misalignment, which complicates 3D interpretation. 3D interpretation is useful for many advanced medical analyses (e.g., apex and valves, accurate definition of wall thickness measurements, improved volume calculations compared to the Simpson method, etc.)

Aligning the slice stack is a non-trivial task. For example, simply stacking the left ventricular pool area (the generally circular shape in the short axis slice) on a common linear axis does not correspond to the real anatomy showing a curved blood pool centerline. The same applies to right ventricle contours that follow a non-trivial curved shape. Thus, a simple registration that maximizes the similarity of the registered slices may not result in an anatomically correct alignment. Furthermore, since slices are typically very thick (8-10mm), the content of consecutive slices can vary considerably, complicating slice-to-slice registration.

US 2017109881 a1 describes aligning short axis images of a heart chamber by performing contour alignment to reduce misalignment between the short axis images. The contour is assumed to follow a quadratic curvature function whose parameters are estimated by minimizing the mean square error. From the estimated center values, the contours are registered using affine transformation with linear interpolation to obtain an aligned contour stack.

Disadvantageously, US 2017109881 a1 assumes that the profile follows a quadratic curvature function, but this is not always the case. For example, the right ventricle contour typically follows a more complex curved shape.

Disclosure of Invention

It would be desirable to have a system and computer-implemented method for slice alignment of a short-axis cardiac Magnetic Resonance (MR) cine playback slice stack that better handles cardiac structures of different shapes.

According to a first aspect of the invention, a system for slice alignment of a short axis cardiac magnetic resonance cine playback slice stack is provided. The system comprises:

an input interface for accessing image data of an input image slice set acquired using a short-axis cardiac magnetic resonance cine playback protocol;

-a processor subsystem configured to:

-accessing training model data defining a machine training model, wherein the machine training model is trained on training data comprising image data of a set of training image slices acquired using a short-axis cardiac magnetic resonance cine-playback protocol, wherein one or more adjacent image slices are mutually misaligned, and wherein the training data further comprises shift values for reducing the mutual misalignment by shifting one or more of the image slices;

-applying the machine training model to a set of neighboring image slices of the input image slice set, thereby obtaining at least one shift value for at least one of the image slices of the set of neighboring image slices; and is

-shifting the image slice based on the shift value.

In accordance with another aspect of the invention, a computer-implemented method for slice alignment of a short axis cardiac magnetic resonance cine playback slice stack is provided. The method comprises the following steps:

-accessing image data of an input image slice set acquired using a short-axis cardiac magnetic resonance cine-playback protocol;

-accessing training model data defining a machine training model, wherein the machine training model is trained on training data comprising image data of a set of training image slices acquired using a short-axis cardiac magnetic resonance cine-playback protocol, wherein one or more adjacent image slices are mutually misaligned, and wherein the training data further comprises shift values for reducing the mutual misalignment by shifting one or more of the image slices;

-applying the machine training model to a set of neighboring image slices of the input image slice set, thereby obtaining at least one shift value for at least one of the image slices of the set of neighboring image slices; and is

-shifting the image slice based on the shift value.

According to another aspect of the invention, there is provided a computer readable medium comprising transient or non-transient data defining a machine training model, wherein the machine training model is configured and trained to be applied to a set of adjacent image slices in a set of image slices acquired using a short axis cardiac magnetic resonance cine-loop protocol, wherein the machine training model is trained to output shift values for reducing mutual misalignment if the set of adjacent image slices are mutually misaligned.

According to another aspect of the invention, there is provided a system for slice alignment of a short axis cardiac magnetic resonance cine playback slice stack comprising:

an input interface for accessing image data representing a set of image slices acquired using a short-axis cardiac magnetic resonance cine playback protocol;

-a processor subsystem configured to:

-accessing surface model data defining a deformable surface model for segmenting cardiac structures in a short axis cardiac MR cine playback slice stack, wherein deformability of the surface model is constrained by shape normalization;

-adapting the surface model to the cardiac structure by detecting boundary points of the cardiac structure in the image data and deforming the surface model towards the boundary points to obtain an adapted surface model adapted in shape to the cardiac structure in the image data; and is

-shifting at least one image slice relative to other image slices such that the boundary points in the image slices obtain an improved match with the cross-sectional representation of the surface model in the respective image slice.

In accordance with another aspect of the invention, a computer-implemented method for slice alignment of a short axis cardiac magnetic resonance cine playback slice stack is provided. The method comprises the following steps:

-accessing image data representing a set of image slices acquired using a short-axis cardiac magnetic resonance cine-loop protocol;

-accessing surface model data defining a deformable surface model for segmenting cardiac structures in a short axis cardiac MR cine playback slice stack, wherein deformability of the surface model is constrained by shape normalization;

-adapting the surface model to the cardiac structure by detecting boundary points of the cardiac structure in the image data and deforming the surface model towards the boundary points to obtain an adapted surface model adapted in shape to the cardiac structure in the image data; and is

-shifting at least one image slice relative to other image slices such that the boundary points in the image slices obtain an improved match with the cross-sectional representation of the surface model in the respective image slice.

According to another aspect of the invention, there is provided a computer-readable medium comprising transient or non-transient data representing a computer program comprising instructions for causing a processor system to perform one or both of the above-described methods.

The above measures provide a solution for slice alignment of short axis cardiac magnetic resonance cine loop slice stacks that does not require additional scans, such as long axis scans or full 3D scans, and is better able to handle cardiac structures having complex shapes. In contrast to US 2017109881 a1, both methods do not require the contour to follow a quadratic curvature function.

In some embodiments, these two methods may be used in concert, where a slice-aligned stack of slices aligned using a deformable surface model may then be purposefully misaligned and used to train a machine-trained model along with shift values representing the misalignment.

Typically, both methods operate on image data of an input image slice set acquired using a short-axis cardiac magnetic resonance cine-loop protocol. Such a set of input image slices is also referred to elsewhere as an image stack or slice stack. As an output, an aligned set of image slices, or at least a better aligned set of image slices, may be obtained, wherein at least one of the image slices is shifted.

Reference is also made to slice alignment using a machine-trained model, noting the following. The machine training model may be configured and trained to be applied to a set of adjacent image slices (e.g. a pair or 3 or 4 adjacent image slices) and to provide at least one shift value for at least one of the image slices as an output, which may be used to improve the mutual alignment of the set of adjacent image slices if the respective image slice is shifted based on the shift value. Such shift values may be expressed in any suitable format, such as a 2D vector defining horizontal and vertical shifts, and may be expressed in various quantities, such as coordinates or pixels in a coordinate system associated with the image stack. In some embodiments, the shift value may define the shift only indirectly, where it may express the misalignment as, for example, a vector, where the shift required to correct the misalignment is equal to subtracting the vector. As such, the term "shift value" should be interpreted as a value indicating the shift to be applied, and the "shift of the image slice based on" the shift value may comprise that a function is applied to the shift value, e.g. reversing the sign of the shift value.

The machine training model may be trained to operate on image data of neighboring image slices themselves. For example, the machine-trained model may be configured and trained to use image intensity values as input, which may be sampled from each of the adjacent slices on the predefined grid (e.g., using 256x256 sample points at known intervals for each slice, e.g., separated by 1 mm). In some embodiments, the machine-trained model may directly use such sampled image intensity values as input, while in other embodiments, the machine-trained model may use differences between image intensity values across image slices as input. Here, "crossing" image slices may refer to calculating the difference between samples located at corresponding positions in each image slice.

Effectively, the above method takes the slice shift estimation as a regression task that is then solved by a machine training model, such as a machine training model based on and trained using deep learning (neural network) techniques.

Reference is also made to slice alignment using deformable surface models, noting the following. Deformable surface models are known per se and may define deformable surfaces for segmenting structures, but which may be constrained in terms of shape deformability by shape normalization. Examples of such deformable surface models are described in a publication entitled "Shape-constrained deformable models and applications in Medical imaging" written by (a subset of) the inventors, at page 151-184 of "Shape Analysis in Medical Image Analysis". The inventors have considered using such a deformable surface model to guide slice alignment. That is, the surface model for the cardiac structure may be adapted to the cardiac structure in a manner known per se, i.e. by detecting boundary points of the cardiac structure in the image data and deforming the surface model towards the boundary points to obtain an adapted surface model adapted in shape to the cardiac structure in the image data. However, due to shape normalization, the deformable surface model is likely to adapt to the general shape of the cardiac structure in the image stack, but is less likely or less adaptable to the typical "zig-zag" pattern in the boundary of the cardiac structure due to misalignment between image slices. After adapting the surface model to the image stack, the adapted surface model can thus be used as a reference for correcting slice misalignments, wherein individual slices can be shifted such that boundary points of cardiac structures detected in the slices better match the cross-sections of the surface model in the respective slice.

Both methods are well suited for the purpose of obtaining a segmentation of the cardiac structure using a deformable surface model. That is, such deformable surface models cannot, but are also not expected to, adapt to "zig-zag" patterns in the boundaries of cardiac structures due to slice misalignment. Such a deformable surface model may better adapt to cardiac structures in the image data after the misalignment between the image slices has been removed or reduced and thus provide a better segmentation of the cardiac structures. Thus, in both cases, the above measures can be followed by applying and adapting a deformable surface model to the image stack, or if such a deformable surface model has been applied and adapted to the image data, by another iteration of adapting the deformable surface model to the image data.

The following optional aspects relate to systems and computer-implemented methods for slice alignment based on machine-trained models. However, where applicable, these optional aspects may refer to corresponding modifications of slice alignment based on deformable surface models.

Optionally, the processor subsystem is configured to:

-applying the machine training model to a set of neighboring image slices of the input image slice set to obtain a series of shift values;

-removing an offset or linear trend from the series of shift values;

-shifting respective image slices of the set of neighboring image slices based on respective shift values of the series of shift values.

The inventors have recognized that the estimation of slice shifts can sometimes be imperfect, and that slice alignment can result in slow translational drift and tilt (e.g., skew) of the overall slice stack, particularly when applied iteratively, e.g., from one slice to the next. This can be avoided or reduced by detecting and then removing the offset or linear trend from the series of shift values before applying the shift values to generate the aligned stack of slices.

Optionally, the machine-trained model is configured and trained to be used as further input position information indicating a position of the respective set of adjacent image slices relative to a cardiac structure shown in the set of input image slices. The slice may show a specific part of the cardiac structure and may therefore have an anatomical position, e.g. a position relative to the cardiac structure. For example, a given slice a may have an anatomical location defined as x% of the range, where 0% is the most apical or even below the apical and 100% is the most basal or above the ventricle. Such anatomical location information may be used as additional input to the machine training model, for example during training and subsequent use, to better account for changes in the input data population.

Optionally, the machine training model is configured and trained to be used as further input angle information indicative of an orientation of a cardiac structure shown in the input image slice set with respect to a coordinate system associated with the input image slice set. Similar to using the anatomical position of the cardiac structure, the orientation of the cardiac structure in the slice stack may be used as an additional input to the machine-trained model, e.g., during training and subsequent use, to better account for variations in the input data population.

Optionally, the processor subsystem is configured to obtain at least one of the position information and the angle information by segmenting the cardiac structure in the input image slice set (e.g. by applying a deformable surface model to the image data). When at least roughly adapted to the image data, the deformable surface model may indicate the position and orientation of the cardiac structure relative to the slice stack. In some embodiments, such a deformable surface model may also present an overall purpose for obtaining a segmentation of the cardiac structure and may have been (at least roughly) adapted to the image data before or during slice alignment.

Optionally, the processor subsystem is configured to mask portions of the image data of the set of input image slices that do not belong to the cardiac structure prior to applying the machine training model to the set of neighboring image slices of the set of input image slices. The slice stack may show anatomical structures other than cardiac structures, such as a chest surrounding a moving heart. Such other anatomical structures may undergo different transformations and thus may interfere with slice alignment. By masking these portions, for example, based on a rough pre-segmentation or localization of the cardiac structure, the machine-trained model can be manipulated during training and subsequent use to focus the slice alignment more on the cardiac structure.

Optionally, the input image slice set is a first image slice set, wherein the input interface is configured for accessing image data in a second image slice set acquired during a different cardiac phase than the first image slice set, and wherein the machine training model is configured and trained to use spatially corresponding samples of the first image slice set and the second image slice set as a joint input. The slice stack may be acquired and may be used for different cardiac phases, e.g., End Diastole (ED) and End Systole (ES) and possibly other cardiac phases. Such a stack of slices may be sampled using the same sampling grid, e.g. using the same acquisition geometry of the MR acquisition apparatus. The intensity values from the two slice stacks may be presented to the machine training model, e.g., during training and subsequent use, as a joint input, e.g., as a vector of two or more intensity values. Thus, one sample input of the machine-trained model may include intensity values from different cardiac phases. Here, a small or no difference between intensity values may represent an area of little motion (or a uniform area), while an area with a large difference in intensity values may indicate motion. This difference may be most prominent in the heart wall and the machine trained model may be manipulated to focus the slice alignment more on the heart structure. Furthermore, by using information of two or more cardiac phases as input to a machine-trained model, the estimation of misalignment by the machine-trained model may be more robust and less sensitive to noise during training and subsequent use.

Optionally, the input image slice set comprises cardiac structure, and wherein the processor subsystem is configured to resample image data of the input image slice set to show the cardiac structure at an anatomical standard position and/or orientation. Such resampling may bring the cardiac structure into an anatomically standard position and/or orientation in the slice stack. Such resampling may be performed explicitly, e.g., to produce a resampled slice stack, or implicitly, e.g., to produce a new sampling grid that may be used to access image data of a slice stack when estimating and performing slice alignment. In the latter case, the new sampling grid may effectively be used as a coordinate transformation, which enables the machine-trained model to access image data of the cardiac structure in a standardized way with respect to anatomical position and/or orientation.

The following optional aspects relate to systems and computer-implemented methods for slice alignment based on deformable surface models. However, where applicable, these optional aspects may refer to corresponding modifications of slice alignment based on machine-trained models.

Optionally, the processor subsystem is configured to deform the surface model towards boundary points of the cardiac structure based on a cost function penalizing a distance of the surface model to the boundary points, and to shift the at least one image slice relative to other image slices such that matching is improved according to the cost function. This type of cost function is known per se and may also be referred to as an "external energy term" or "data fitting term".

Optionally, the processor subsystem is configured for iterative slice alignment by repeating the adaptation of the surface model and the shifting of the at least one image slice at least twice. The adaptation of the surface model and the slice alignment based on the adapted surface model may be performed iteratively, wherein these two steps may be repeated at least twice. As the slice misalignment is gradually reduced, the surface model may gradually adapt to better fit the image data.

Those skilled in the art will appreciate that two or more of the above-described embodiments, implementations, and/or optional aspects of the invention may be combined in any manner deemed useful.

Modifications and variations of the system, the computer-implemented method, and/or any computer program product, which correspond to the described modifications and variations of another of the entities, may be performed by a person skilled in the art based on the current description.

Drawings

These and other aspects of the invention will be apparent from and elucidated further with reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which

Fig. 1 shows a system for slice alignment of short axis cardiac magnetic resonance cine loop slice stacks that may use a machine training model in one embodiment and a deformable surface model in another embodiment;

figure 2 illustrates a system for training a machine trainable model of slice alignment for short axis cardiac magnetic resonance cine playback slice stacking;

FIG. 3 shows a cross-section of a misaligned slice stack showing boundary points of a cardiac structure and an applied deformable surface model;

FIG. 4 shows the slice stack of FIG. 3 after shifting slices to align boundary points with the applied deformable surface model;

FIG. 5 shows a set of neighboring slices and sampling grids that indicate which intensity values are used as inputs to a machine training model;

fig. 6 shows neighboring slices of two slice stacks acquired during different cardiac phases, wherein a sampling grid is defined in the two slice stacks;

FIG. 7 shows a cross-section of image slices of a cardiac structure and an applied deformable surface model, the model shown misaligned with a boundary of the cardiac structure due to misalignment between the image slices;

FIG. 8 shows the slice stack of FIG. 7 after shifting the image slice;

FIG. 9 illustrates a computer-implemented method for slice alignment using a machine-trained model;

FIG. 10 illustrates a computer-implemented method for slice alignment using a deformable surface model; and is

FIG. 11 illustrates a computer-readable medium including data.

It should be noted that the figures are purely diagrammatic and not drawn to scale. In the drawings, elements corresponding to elements already described may have the same reference numerals.

List of reference numerals

The following list of reference numerals is provided to facilitate the explanation of the figures and should not be construed as limiting the claims.

020. 022 data storage device

030. 032 short axis cardiac MR cine playback slice Stack

040 displacement value

050 trained model data

060 surface model data

100 system for slice alignment

120 input interface

122. 124 data communication

140 processor subsystem

160 communication interface

200 System for training a model for slice alignment

220 input interface

222. 224 data communication

240 processor subsystem

260 communication interface

300 misaligned short axis cardiac MR cine playback slice stack

302 aligned short axis cardiac MR cine playback slice stack

310 parts of cardiac structures

320 structural wall of heart

360. 362 surface of deformable surface model

S1-S3 applied to shifting of image slices

400 set of adjacent image slices from a given cardiac phase

402 set of adjacent image slices from another cardiac phase

410-412 image slices

420-424 sampling grid defining inputs to a machine training model

Image slice before 500 slice alignment

502 slice aligned image slice

560 surface of deformable surface model

600 method for slice alignment using a machine training model

610 access image data of a slice stack

620 access data of machine training model

630 applying machine training models to adjacent slices

640 shifting image slice(s)

700 method for slice alignment using surface models

710 accessing image data of a slice stack

720 access data of a deformable surface model

730 adapting a surface model to a cardiac structure

740 shifting image slice(s)

750 repeat iterations

800 computer readable medium

810 non-transient data

Detailed Description

Fig. 1 shows a system 100 for slice alignment of short axis cardiac magnetic resonance cine loop slice stacking that may use a machine trained model in one embodiment and a deformable surface model to perform slice alignment in another embodiment. The system 100 is shown to include an input interface 120 for accessing image data 030 of a set of input image slices acquired using a short-axis cardiac magnetic resonance cine-playback protocol. For example, as also shown in fig. 1, the input interface 120 can provide data access 122 to an external data storage device 020 that can include the image data 030. Alternatively, the input interface 120 may provide data access to an internal data storage device that is part of the system 100. Alternatively, the image data 030 may be accessed via a network. Generally, the input interface 120 may take various forms, such as a network interface to a local or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, and so forth. The data storage device 020 may take any known and suitable form.

The data storage device is also shown to include training model data 050 and surface model data 060, which will be explained further below. According to embodiments, the data storage device may comprise one or both types of data 050, 060. In some embodiments, image data 030, training model data 050, and surface model data 060 may each be accessible from a different data storage device.

System 100 is also shown to include a processor subsystem 140 that may be in internal communication with input interface 120 via data communication 124, and as an optional component, with a communication interface 160. As also shown in fig. 1, communication interface 160 may be an external communication interface, such as a network interface to a local area network or a wide area network (e.g., the internet). In some embodiments, input interface 120 may be the same interface as communication interface 160. In other embodiments, the communication interface 160 may be a network interface via which data is received, and the input interface 120 may be a data storage interface via which data is stored.

In one embodiment, the processor subsystem 140 may be configured to access training model data 050 that defines a machine training model trained on training data comprising image data of a set of training image slices acquired using a short-axis cardiac magnetic resonance cine-playback protocol. Such a machine training model may be obtained, for example, from the system described with reference to fig. 2. The processor subsystem 140 may be configured to apply the machine-trained model to a set of neighboring image slices of the input image slice set, thereby obtaining at least one shift value for at least one of the image slices of the set of neighboring image slices, and shift the image slice based on the shift value. Various details and aspects of this embodiment, including optional aspects thereof, will be further elucidated with reference to, inter alia, fig. 5 and 6.

In another embodiment, the processor subsystem 140 may be configured to access surface model data 060 defining a deformable surface model for segmenting cardiac structures in a short axis cardiac MR cine playback slice stack. The deformability of the surface model may be constrained by shape normalization. Processor subsystem 140 may be configured to: adapting the surface model to the cardiac structure by detecting boundary points of the cardiac structure in the image data and deforming the surface model towards the boundary points to obtain a shape adapted to the cardiac structure in the image data; and shifting at least one image slice relative to the other image slices such that boundary points in the image slice obtain an improved match with the cross-sectional representation of the surface model in the respective image slice. Various details and aspects of this embodiment, including optional aspects thereof, will be further set forth in this specification with reference to fig. 3 and 4.

In general, the system 100 may be embodied as or in a single device or apparatus, such as a workstation, e.g., based on a laptop or desktop computer, or a server. The apparatus or device may comprise one or more microprocessors executing appropriate software. For example, a processor subsystem may be embodied by a single Central Processing Unit (CPU), but may also be embodied by a combination or system of such CPUs and/or other types of processing units. The software may have been downloaded and/or stored in a corresponding memory, e.g. a volatile memory such as a RAM or a non-volatile memory such as a flash memory. Alternatively, the functional elements of the system, such as the input interface and the processor subsystem, may be implemented in the form of programmable logic in a device or apparatus, such as a Field Programmable Gate Array (FPGA). Generally, each functional unit of the system may be implemented in the form of a circuit. It should be noted that the system 100 may also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as a distributed server, e.g., the system 100 may also be implemented in the form of cloud computing.

Fig. 2 illustrates a system 200 for training a machine trainable model of slice alignment for short axis cardiac magnetic resonance cine playback slice stacking. System 200 is shown to include an input interface 220 and a processor subsystem 240 that communicate via a data communication 224. System 200 is also shown to include a communications interface 260. Input interface 220 is shown providing access to a data storage device 022 via data communication 222. The components of the system 200 described thus far may correspond in general type to the corresponding components of the system 100 of fig. 1. In particular, the embodiment options described in the previous paragraph may also be applicable to the system 200.

However, unlike the system 100 of fig. 1, the processor subsystem 240 of the system 200 is configured to train a machine trainable model, i.e., using training data comprising image data 032 of a set of training image slices acquired using a short axis cardiac magnetic resonance cine-loop protocol, and wherein one or more adjacent image slices are misaligned with respect to each other. Further, the training data used for training may comprise shift values 040 for reducing the mutual misalignment by shifting one or more of the image slices. The processor subsystem 240 may be configured to train the machine trainable model using the training data to obtain a machine training model, and in particular to obtain training model data 050 representing the machine training model. The machine trainable model may be of any suitable type, such as a neural network, for example a Deep Neural Network (DNN) comprising one or more convolutional layers (convolutional DNNs). Such neural networks may be trained using any suitable machine learning technique.

Figure 3 shows a cross-section of a misaligned slice stack 300 that may be obtained by acquisition using a short-axis cardiac magnetic resonance cine-playback protocol. It can be seen that in such a stack of slices, the boundaries of (parts of) the cardiac structure 310 may be misaligned between the individual slices in a "zigzag" pattern, which is visible in fig. 3 by the wall 320 (in this example the myocardial wall) of the cardiac structure 310 following the "zigzag" pattern. Such a misalignment may hinder adaptation of the deformable surface model to the image data and thus segmentation of the cardiac structure 310. That is, such deformable surface models may follow the general boundaries of the cardiac structure 310 when adapted to image data, but may prevent following an irregular "zigzag" misalignment pattern altogether due to shape normalization. Fig. 3 does illustrate that the surfaces 360, 362 of the deformable surface model may follow the general boundaries of the cardiac structure 310, but may often be misaligned with the wall 320 in individual slices due to the "zig-zag" misalignment pattern described above. At the same time, the "zigzag" pattern may confuse finer anatomical details of the cardiac structure 310 and thereby prevent or hinder the deformable surface model from further adapting to such finer anatomical details.

The system 100 described with reference to fig. 1 may in an embodiment perform slice alignment by adapting a deformable surface model to the boundary of the cardiac structure, e.g. as illustrated in fig. 3, and then shift individual slices of the slice stack such that boundary points comprised in the respective slice better match the cross-section of the hitherto adapted deformable surface model in the slice, e.g. as illustrated in fig. 4 by S labeled as referring to the direction of the respective slice shift₁-S₃Is illustrated by the arrows of (a). It should be noted that although fig. 3 and 4 do not explicitly show such boundary points, these may be detected in a manner known per se and may indicate to the system 100 where the boundary of the cardiac structure in each slice is located, here the wall 320. Effectively, at this step in slice alignment, individual slices can be adapted to the deformable surface model (by appropriate displacement), but not vice versa. The entire slice alignment process may be performed iteratively, wherein the deformable surface model may be adapted to the slice stack, then the individual slices may be adapted to the deformable surface model (by appropriate shifting), after which the deformable surface model may be adapted again to the slice stack, etc. Can be used forTo repeat the process, e.g., a fixed number of times or until a convergence criterion is reached.

With continued reference to fig. 3 and 4, it is noted that the application of a deformable surface model to the image data for segmentation purposes is also referred to as model-based image segmentation. Model-based image segmentation typically uses triangulated surface meshes, and surface meshes may be applied to image data by iteratively detecting boundary points in the image (e.g., each triangle) and by searching for boundaries along the triangle normals and then adapting the surface meshes to the detected boundaries. The shape model may normalize mesh deformations. Surface model based slice alignment may be based on that the surface model may roughly interpolate the view through the original slice stack. Due to shape model normalization, an accurate fit to the "zigzag" boundaries of the cardiac structure may not be achieved. The methods shown in fig. 3 and 4 may add new steps to this iterative adaptation. That is, after the surface mesh has achieved an intermediate fit to the slice stack, the adaptive strategy may change: rather than attract the triangles of the surface mesh to the detected boundary points, the individual slices are moved with their boundary points towards the surface mesh. To improve the overall result, mesh adaptation and slice shifting may alternate in a configurable scheme. For example, after the start condition is met, the slice may be shifted using every second iteration, while further mesh adaptation may be performed using every other iteration. This may improve slice alignment and may improve segmentation of cardiac structures, since better aligned slices may lead to improved boundary detection (due to interpolation between slices during boundary detection) and thus better mesh adaptation.

In a particular embodiment, boundary detection for a triangle may be limited to searching for boundary points in the same slice, e.g., to avoid shifting slices in planes "above or below" away from the corresponding triangle. Disparity vectors pointing from triangles to boundaries may also need to be reasonably parallel to the slice, e.g., as quantified by a correspondence metric, rather than pointing primarily along the stacking direction of the slice. This may improve the numerical stability of the slice alignment. Moreover, boundary detection may be configured to reject suspicious boundaries, such that outliers may be filtered out.

As previously mentioned, fig. 4 shows the slice stack of fig. 3 after shifting several slices into alignment with the applied deformable surface model.

It should be noted that even though the adaptive surface model may generally follow the cardiac structure in a non-aligned slice stack, and the estimated slice shifts are expected to add up roughly to zero, and thus do not introduce any tilt of the slice stack, the actual slice shift estimation may not be perfect. Thus, estimating and applying the slice shift over multiple iterations may result in slow translational drift and/or tilt (skew) of the slice stack, and the surface model may follow such drift when iteratively adapting the surface model to the slice stack. To compensate for such drift and/or tilt, an additional slice shift normalization step may be introduced, which may include the following (described with reference to the x-component of the shift; the y-component may be normalized accordingly, where x and y refer to the intra-slice coordinate system): let { dx [ z ]]Is the estimated x displacement of all slices indexed by z. The "normalized" displacement of the linear transform can be defined as { T (dx [ z ]]Z), wherein T (dx [ z ]) is],z)＝dx[z]+ a.z + b, where the linear transformation parameters a and b can be estimated such that ∑ is_Z∑_ZT(dx[z]Z) is minimized, e.g., so that T (dx [ z ]]Z) deviates as little as possible from a straight line along the slice normal (in the z direction) centered at x-0. This also means ∑_ZT(dx[z]And z) is 0, thereby eliminating translational drift. Such normalization may thus remove an offset or linear trend from the sequence of shift values representing the shift values of the sequence of image slices.

Referring also to slice alignment using machine-trained models such as convolutional DNN, the method treats slice shift estimation as a regression task solved using machine learning techniques. Here, pairs or n-tuples of consecutive slices may be input into a machine training model that derives the relative shift that will bring the slice to an auto-alignment position based on the image intensity values of the slice and, in some embodiments, based on auxiliary information. To train such a model, a stack of slices aligned by the method described with reference to fig. 3 and 4 may be used as a reference. For example, such a stack of slices may be purposefully misaligned using known shift values, and the purposefully misaligned stack of slices may be used with the known shift values as training data for a machine trainable model to learn predicted slice shifts.

FIG. 5 shows a set of neighboring slices 400 and a sampling grid 420 and 422 that indicates which intensity values from the individual image slices 410 and 412 are used as inputs to the machine training model. For example, the sampling grid may be a predefined grid, for example, using 256x256 sampling points at known intervals (e.g., 1mm apart). For example, when the intensity values of three adjacent slices are input to a machine training model, the model may be fed with 3x256x256 intensity values. Various other sampling grids may also be defined, e.g., involving different numbers of samples, different spatial shapes, different numbers of slices, etc. In some embodiments, the difference in intensity values between slices may be used as an input. The intensity values may be normalized, for example using statistics from the complete stack, such as the mean and standard deviation or percentile of the intensity.

For an example of pairwise estimation of slice shifts, note the following. For N slices, a global shift may be applied to all slices without changing relative alignment or misalignment. However, the relative position between the N slices can be defined by N-1 relative shifts. Such a relative shift may be expressed in various ways, for example, as a shift of each slice relative to a reference slice (e.g., the first or last slice in a slice stack), or as a relative shift between consecutive slices i and i + 1. In summary, N-1 relative slice shifts of N slices can thus be estimated. For one of the slices (e.g., the first or last slice), a zero shift may be assigned. However, this may result in an overall shift of the estimated relative shifts of the other N-1 slices, since these sums may not be zero. Furthermore, as also previously described, slice shift estimation may be imperfect and introduce drift and/or tilt. The previously described slice shift normalization step may thus be applied to a series of shift values obtained by slice shift estimation to remove bias, e.g. offset (global shift described above) or linear trend, from the shift values to minimize deviation from the z-axis (referring to the axis orthogonal to the in-slice axis).

In addition to image intensity, various other types of auxiliary information and inputs to machine training/trainable models may be used. For example, the anatomical content of a slice may vary from apex to "base" (transition from ventricle to atrium), and thus vary considerably throughout the slice stack. Machine-trained/trainable models may benefit from knowing that "more vertices" or "more bases" slices must be aligned. Furthermore, since slices may show the heart with different rotations, angular information may also be helpful.

The machine-trained/trainable model is referred to below as a neural network, but is applicable to other types of machine-trained/trainable models.

There are various ways to provide this additional location information to the neural network. In one embodiment, the location information may be provided to the neural network by indicating that a particular slice is at a particular relative location in the slice stack, for example, by specifying a percentage where 0% is the most apical or even below apical and 100% is the most basal or above ventricular. In another embodiment, 3D coordinates may be specified for each sample point. For example, the acquired DICOM information may be used such that the relative position with respect to the center of the volume may be specified in so-called "patient" (or "world") coordinates, where "z" is from foot to head, "x" is from right to left, and "y" is from front to back of the patient. Since the heart has a typical orientation within each patient, such DICOM-based location information may already provide a rough indication of the anatomical content of a particular slice/sample point. However, both the size and precise orientation of the heart vary from patient to patient, so that a more specific coordinate system may be helpful. In a more detailed embodiment, approximate segmentation of the heart (e.g., with imperfect or no slice alignment) may be used to define a coordinate system based on cardiac landmarks, e.g., also for defining 2-, 3-, and 4-chamber plus short-axis views. Since this may require a complete cardiac segmentation, an intermediate embodiment may use a coordinate system estimated by the Generalized Hough Transform (GHT), which may be applied in various scales and various rotations of its edge templates. Anatomical location information for the slice or sample point can then be determined in the coordinate system and provided as input to the neural network.

Various types of pre-processing may also be used before or after training using the machine training model. This may reduce the complexity of the learning problem and incorporate prior knowledge into the input to the machine training model. For example, the image data of the slice stack may be resampled in a new (anatomically aligned) coordinate system such that the cardiac anatomy follows a standard orientation in that coordinate system. In addition to sampling the image intensity on a grid aligned with the scan axis, the sampling grid can be reoriented in each individual slice using an axis associated with the anatomical direction. The only additional location information input to the neural network may be the z-coordinate encoding the anatomical location of the slice.

Additionally or alternatively, the coarse pre-segmentation or localization may be used to mask irrelevant parts of the image data, such as the thorax, which may present a different transformation compared to a moving heart.

Knowledge of the orientation of the anatomical structure can also be used in a cyclic architecture, which predicts a series of translations when the input consists of a series of consecutive (or even a single) slices, for example always starting close to the apex and ending around the basal part of the heart. This makes it possible to use the position information along the long axis of the heart even if there is no correlation between the transitions from slice to slice. That is, the internal state may be updated at each new slice, but the information of the previous slice may be preserved. Thus, the neural network may know the current slice position, which may cause the output portion of the neural network to act differently. In this way, typical alignment curves across the population may be implicitly considered. This method may also utilize a so-called "dual stream" in which upward (from the apex) and downward (from the base portion) passes may be combined.

Fig. 6 shows two sets 400, 402 of adjacent image slices from two slice stacks each acquired during a different cardiac phase, and wherein sampling grids 420, 424 are defined in the two sets of adjacent image slices. That is, the image intensities from different cardiac phases may be used as joint inputs to the neural network. In the example of fig. 6, intensity values from two cardiac phases (e.g., ED and ES) may be sampled on the same grid in each cardiac phase, e.g., using sampling grids 420 and 424. In this way, each sample point used as input to the neural network may provide an intensity value from both cardiac phases.

Fig. 7 shows a cross-section of an image slice 500 of a cardiac structure and an adapted deformable surface model, and in particular a cross-section of a surface 560 of the deformable surface model that is misaligned with respect to a boundary of the cardiac structure due to misalignment between the image slices. Fig. 8 shows the result of slice alignment as described in this specification, where the image slice of fig. 7 is shown after shifting the image slice within the slice stack, and the deformable surface model is shown after another iteration of adapting the deformable surface model to the slice stack. This may result in improved alignment of the surface 500 with the boundary of the cardiac structure in the image slice 502.

In general, slice alignment as described in this specification can be used in various medical systems and devices, including but not limited to 3D visualization systems, such as the "orthoviewer" visualization system, which shows two-and four-chamber multi-planar formats in addition to individual slices.

FIG. 9 shows a flow diagram of a computer-implemented method 600 for slice alignment using a machine-trained model. The method 600 may correspond to the operation of the system 100 of fig. 1 when configured for slice alignment using a machine-trained model. However, this is not a limitation, wherein method 600 may also be performed using another system, apparatus, or device.

The method 600 may include, in an operation entitled "accessing image data for a slice stack," accessing 610 image data for a set of input image slices acquired using a short-axis cardiac magnetic resonance cine-loop protocol. The method 600 may further include, in an operation entitled "accessing data for a machine training model", accessing 620 training model data defining the machine training model, wherein the machine training model is trained on training data comprising image data of a set of training image slices acquired using a short-axis cardiac magnetic resonance cine-loop protocol, wherein one or more adjacent image slices are mutually misaligned, and wherein the training data further comprises shift values for reducing the mutual misalignment by shifting one or more of the image slices. The method 600 may further include, in an operation entitled "applying a machine training model to neighboring slices", applying 630 the machine training model to a set of neighboring image slices in the input image slice set to obtain at least one shift value for at least one image slice in the set of neighboring image slices, and, in an operation entitled "moving image slices", shifting 640 the image slices based on the shift values.

FIG. 10 shows a flow diagram of a computer-implemented method 700 for slice alignment using a machine-trained model. The method 700 may correspond to the operation of the system 100 of fig. 1 when configured for slice alignment using a deformable surface model. However, this is not a limitation, wherein method 700 may also be performed using another system, apparatus, or device.

The method 700 may include, in an operation entitled "accessing image data for a slice stack," accessing 710 image data in an image slice set representing acquisition using a short-axis cardiac magnetic resonance cine-loop protocol. The method 700 may further include, in an operation entitled "accessing data for a deformable surface model", accessing 720 surface model data defining the deformable surface model for segmenting cardiac structures in a short axis cardiac MR cine playback slice stack, wherein a deformability of the surface model is constrained by shape normalization. The method 700 may further comprise, in an operation entitled "adapting a surface model to a cardiac structure", adapting 730 the surface model to the cardiac structure by detecting boundary points of the cardiac structure in the image data and deforming the surface model towards the boundary points to obtain an adapted surface model, the adapted surface model being adapted in shape to the cardiac structure in the image data. The method 700 may further include, in an operation entitled "shift image slice(s)", shifting 740 at least one image slice relative to other image slices such that boundary points in the image slice obtain an improved match to the cross-sectional representation of the surface model in the respective image slice. Operations 730, 740 may be repeated in iteration 750.

It will be appreciated that, in general, the operations of method 600 of fig. 9 and/or method 700 of fig. 10 may be performed in any suitable order, e.g., sequentially, simultaneously, or in combinations thereof, subject to a particular order, e.g., by input/output relationships, where applicable.

The method(s) may be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. As also illustrated in fig. 11, instructions for a computer, e.g., executable code, may be stored on a computer-readable medium 800, e.g., in the form of a series of machine-readable physical markings 810 and/or as a series of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code may be stored in a transient or non-transient manner. Examples of computer readable media include memory devices, optical storage devices, integrated circuits, servers, online software, and so forth. Fig. 11 shows an optical disc 800. Alternatively, the computer-readable medium 800 may include transient or non-transient data 810 representing a machine training model as described elsewhere in this specification.

The examples, embodiments, or optional features, whether or not indicated as non-limiting, are not to be construed as limiting the claimed invention.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The use of the word "comprise" and variations such as "comprises" or "comprising" does not exclude the presence of elements or steps other than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one" when preceded by a list or group of elements indicate a selection of all elements or any subset of elements from the list or group. For example, the expression "at least one of A, B and C" should be understood to include only a, only B, only C, A and both B, both a and C, both B and C, or all A, B and C. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.