阅读说明：本技术 一种透明桥和非透明桥功能可选的pcie交换器及多主机系统 (PCIE exchanger with optional functions of transparent bridge and non-transparent bridge and multi-host system ) 是由 刘威 龚锐 石伟 周宏伟 张剑锋 任巨 杨乾明 张见 王永文 于 2020-06-18 设计创作，主要内容包括：本发明公开了一种透明桥和非透明桥功能可选的PCIE交换器及多主机系统,PCIE交换器包括透明桥、第一选择器、第二选择器、第三选择器以及非透明桥,第一选择器的固定端口作为上游端口,第一选择器的选择端口分别与透明桥的上游端口、第二选择器的一个选择端口相连,第二选择器的选择端口与透明桥的下游端口相连,第二选择器的固定端口分为两路且其中一路串接非透明桥后与第三选择器的一个选择端口相连、另一路作为非透明桥的旁路通道与第三选择器的另一个选择端口相连,第三选择器的固定端口作为PCIE交换器的一个下游端口。本发明能够实现透明桥和非透明桥功能可选,实现地址路由和ID路由兼容。(The invention discloses a PCIE exchanger with selectable functions of a transparent bridge and a non-transparent bridge and a multi-host system, wherein the PCIE exchanger comprises the transparent bridge, a first selector, a second selector, a third selector and the non-transparent bridge, a fixed port of the first selector is used as an upstream port, a selection port of the first selector is respectively connected with an upstream port of the transparent bridge and a selection port of the second selector, a selection port of the second selector is connected with a downstream port of the transparent bridge, a fixed port of the second selector is divided into two paths, one path of the two paths is connected with one selection port of the third selector after being connected with the non-transparent bridge in series, the other path of the two paths is connected with the other selection port of the third selector as a bypass channel of the non-transparent bridge, and a fixed port of the third selector is used as a downstream port of the PCIE exchanger. The invention can realize the function selection of the transparent bridge and the non-transparent bridge and realize the compatibility of the address route and the ID route.)

1. A PCIE switch with selectable transparent bridge and non-transparent bridge functions comprises a transparent bridge, characterized in that the device also comprises a first selector, a second selector, a third selector and a non-transparent bridge, the fixed port of the first selector is used as the upstream port of the PCIE exchanger, the selection port of the first selector is respectively connected with the upstream port of the upstream PCI-PCI bridge of the transparent bridge and one selection port of the second selector, the select port of the second selector is coupled to a downstream port of a downstream PCI-PCI bridge of the transparent bridge, the fixed port of the second selector is divided into two paths, one path of the fixed port is connected with one selection port of the third selector after being connected with the non-transparent bridge in series, the other path of the fixed port is used as a bypass channel of the non-transparent bridge and is connected with the other selection port of the third selector, and the fixed port of the third selector is used as a downstream port of the PCIE exchanger.

2. The PCIE switch with selectable transparent and non-transparent bridge functions of claim 1, wherein the non-transparent bridge comprises an upstream endpoint device module UEP, a downstream endpoint device module DEP, a doorbell register module, and a scratch register module, and four lanes are connected in parallel between the upstream endpoint device module UEP and the downstream endpoint device module DEP: the first path comprises an outbound address conversion module and an outbound request message ID conversion module; the second path comprises a message ID conversion module which responds to the outbound request and completes the message ID conversion; the third path comprises a message ID conversion module for responding the inbound request; the fourth path comprises an inbound address conversion module and an inbound request message ID conversion module; the upstream end point device module UEP is connected with the fixed port of the second selector in the upstream direction, the downstream direction is respectively connected with the outbound address conversion module, the outbound request response completion message ID conversion module, the inbound request message ID conversion module, the doorbell register module and the note register module, and the upstream end point device module UEP and the downstream end point device module DEP are standard PCIE end point devices.

3. The PCIE switch with optional transparent bridge and non-transparent bridge functions of claim 2, wherein the upstream endpoint device module UEP has a plurality of base address registers therein, and is configured to allocate configuration, IO addresses and memory spaces in the first system connected to the upstream port of the PCIE switch; the downstream endpoint device module DEP is provided with a plurality of base address registers and is used for allocating configuration, IO addresses and memory spaces in a second system connected to an upstream port of the PCIE switch; the outbound address conversion module and the inbound address conversion module are used for realizing the address space conversion between the IO address and the memory space between the first system and the second system, the outbound request message ID conversion module and the inbound request message ID conversion module are used for realizing the request message ID conversion between the first system and the second system, and the outbound request response completion message ID conversion module and the inbound request response completion message ID conversion module are used for realizing the ID conversion between the outbound request response completion message and the inbound request response completion message between the first system and the second system.

4. The PCIE switch with selectable transparent bridge and non-transparent bridge functions of claim 3, wherein the detailed step of the outbound address translation module implementing the address space translation of the IO address and the memory space between the first system and the second system includes: the outbound address conversion module carries out base address matching in a base address register on an address field of a received request message from a first system, if the address field of the received request message is matched with the base address of the first system, the offset of the message address field and the base address of the base address conversion register are recombined into an address in a second system address field, the base address in the second system field is stored in the base address conversion register, and if the address field of the received request message is not matched with the base address of the base address conversion register, the outbound address conversion module directly discards or indicates an upstream endpoint device module UEP to return to a state that the request is not supported; the detailed steps of the inbound address translation module for implementing address space translation between the IO address and the memory space of the first system and the second system include: the inbound address translation module matches the base address in the base address register for the address field of the received request message from the second system, if the address field matches the base address, the offset of the message address field and the base address of the base address translation register are recombined into the address in the first system address field, the base address in the first system field is stored in the base address translation register, and if the address field does not match the base address, the inbound address translation module directly discards or instructs the downstream endpoint device module DEP to return to the address field that does not support the request.

5. The PCIE switch with selectable transparent bridge and non-transparent bridge functions according to claim 3, wherein the outbound request packet ID conversion module has a global table, the global table needs to be initialized after the first system enumeration configuration is completed and before the first system sends out the request, the table needs to design 8 entries, each entry has 4 fields: the system comprises an INDEX field, a Bus field, a Dev field and a Func field, wherein the Func field is a function number of a first system domain, the INDEX field is a function number of a second system domain corresponding to the function number of the first system domain, the Bus field is used for recording a Bus number, the Dev field is used for recording a device number, and the lengths of the Dev field and the Dev field are 3 bits, 8 bits, 5 bits and 3 bits respectively; after receiving a first system domain request message ID = { FBus, FDev, FFunc }, an outbound request message ID conversion module, wherein the FBus represents a bus number, the FDev represents an equipment number, and the FFunc represents a function number; firstly, a global table is searched for by adopting a content matching search mode according to a request ID of a first system domain request message, FBus is used for matching a Bus field, FDev is used for matching a Dev field, FFunc is used for matching a Func field, and if any item in the global table is not hit, the request is returned to a state that the request is not supported; replacing the FFunc field of the request ID with the INDEX field of the hit table entry, so that the INDEX field of the hit table entry is a new FFunc field of the converted request ID of the second system request message, replacing the FDev field of the request ID with the device number DEPDev of the second system captured by the downstream endpoint device module DEP, and replacing the FBus field of the request ID with the device number DEPBus of the second system captured by the downstream endpoint device module DEP, so that the request message ID = { DEPBus, DEPDev, INDEX } converted to the second system domain is obtained.

6. The PCIE switch with optional transparent bridge and non-transparent bridge functions of claim 5, wherein when the response inbound request completion packet ID conversion module implements ID conversion of the response inbound request completion packet between the first system and the second system, the response inbound request completion packet ID = { DEPBus, DEPDev, INDEX } from the second system domain is received, the INDEX field therein is used to access the global table of the outbound request packet ID conversion module, the Bus field, the Dev field, and the Func field of the hit entry are read out and filled in the converted response inbound request completion packet ID = { Bus, Dev, Func } entering the first system.

7. The PCIE switch with optional transparent bridge and non-transparent bridge functions as recited in claim 3, wherein the inbound request packet ID conversion module has a global table, the global table needs to be initialized after the first system enumeration configuration is completed and before the second system sends out the request, the table needs to design 8 entries, each entry has 4 fields: the system comprises an INDEX field, a Bus field, a Dev field and a Func field, wherein the Func field is a function number of a second system domain, the INDEX field is a function number of a first system domain corresponding to the function number of the second system domain, the Bus field is used for recording a Bus number, the Dev field is used for recording a device number, and the lengths of the Dev field and the Dev field are 3 bits, 8 bits, 5 bits and 3 bits respectively; after receiving a second system domain request message ID = { FBus, FDev, FFunc }, an outbound request message ID conversion module, wherein the FBus represents a bus number, the FDev represents an equipment number, and the FFunc represents a function number; firstly, a global table is searched for a request ID of a second system domain request message in a content matching search mode, FBus is used for matching a Bus field, FDev is used for matching a Dev field, FFunc is used for matching a Func field, and if any item in the global table is not hit, the request is returned to a state that the request is not supported; replacing the FFunc field of the request ID with the INDEX field of the hit table entry, so that the INDEX field of the hit table entry is a new FFunc field of the request ID of the converted first system request message, replacing the FDev field of the request ID with the device number DEPDev of the first system captured by the upstream endpoint device module UEP, and replacing the FBus field of the request ID with the device number DEPBus of the first system captured by the downstream endpoint device module DEP, so that the request message ID = { DEPBus, DEPDev, INDEX } converted to the first system domain is obtained.

8. The PCIE switch as claimed in claim 7, wherein when the outbound request completion packet ID conversion module implements ID conversion of the outbound request completion packet between the first system and the second system, the inbound request completion packet ID = { debpus, depdevev, INDEX } from the first system domain is received, the INDEX field in the inbound request packet ID conversion module is used to access the global table of the inbound request packet ID conversion module, the Bus field, the Dev field, and the Func field in the hit table are read out, and the converted inbound request completion packet ID = { Bus, Dev, Func } entering the second system is filled in.

9. The PCIE switch with selectable transparent bridge and non-transparent bridge functions of claim 2, wherein the doorbell register module comprises a set of registers for passing interrupts between the first system and the second system, the registers of the doorbell register module comprising: the system comprises a first system interrupt state register, a first system interrupt request register, a first system interrupt masking set register, a first system interrupt masking clear register, a second system interrupt state register, a second system interrupt request register, a second system interrupt masking set register and a second system interrupt masking clear register, wherein each register can be accessed by the first system and the second system through a memory address or IO address space, and as long as the request register has a position set and is not masked, a valid interrupt is output; if the request bit is cleared or masked, the interrupt is invalidated; the note register module also comprises a group of registers, the first system and the second system can access the note register module through a memory address or IO address space and can be read and written, and the group of registers are used for transmitting control and state information or are used as read-write state registers.

10. A multi-host system, at least comprising a first system and a second system, wherein the first system and the second system are connected by a PCIE switch, and the PCIE switch is the PCIE switch with the selectable functions of the transparent bridge and the non-transparent bridge according to any one of claims 1 to 9.

Technical Field

The invention relates to the field of integrated circuit design, in particular to a PCIE exchanger with optional functions of a transparent bridge and a non-transparent bridge and a multi-host system.

Background

PCI Express (Peripheral Component interconnect Express, PCIE for short) is a high-speed serial computer expansion bus standard, and PCIE devices can be divided into three types: root complex, SWITCH, and Endpoint device (Endpoint). A typical PCIE tree topology is shown in fig. 1, where there is only one root complex in the tree topology, and it is responsible for discovering the structure of the whole topology, including buses and node devices therein, and allocating corresponding bus numbers and address spaces. PCIE SWITCH contains a number of PCI-PCI bridges within it, which are commonly referred to as transparent bridges.

In recent years, distributed systems have been widely developed, and a multi-host system can provide high bandwidth and better stability. However, the provision of only one root complex in the tree topology in the PCIE protocol makes it impossible for the conventional PCIESWITCH to easily construct a multi-host system, and thus a non-transparent bridge needs to be implemented in the SWITCH to isolate the address spaces of the multi-host system. The non-transparent bridge is assumed to be an Endpoint, and the Type0 configuration space is exposed to the two host ends, so that enumeration software on the two sides can regard the non-transparent bridge as a leaf node in the topology, that is, the two host systems cannot see the device in the opposite topology, thereby achieving the purpose of being isolated. Meanwhile, the nontransparent bridge also performs address and ID conversion on bidirectional requests by using a BAR register in a configuration header of Type0, converts the initiators of the requests from the host system into the nontransparent bridge, and then performs routing in the other host system, thereby realizing address space sharing between the two host systems.

The PCIE protocol defines three routing modes: address routing; ID routing and implicit routing. Where ID means the node number, consisting of the BUS number, DEV number and FUNC number of the node, also commonly referred to as BDF. The PCIE protocol also defines four transactions: storing and reading; IO reading and writing; configuring reading and writing; a message. MEM read and write and IO read and write are address routes, configuration read and write and completion messages are ID routes, and messages are usually implicit routes. The non-transparent bridge must support the conversion processing of the above four messages to realize the interaction between the two host systems. In the system architecture of the SWITCH non-transparent bridge, the host system can be connected to a port of the non-transparent bridge. The application scenario of the architecture can be that two host systems exchange memory data, or the host system with a non-transparent port exchanges data with a device in another address domain. The two application scenarios have different requirements on the system architecture, and data is exchanged with devices that are not on the topology tree, so that a path for accessing the devices is necessary, that is, the converted request can also be sent to a virtual bus inside the SWITCH. These accesses are also delay sensitive, which of course is more prominent in memory data exchange applications.

In summary, the structure of the non-transparent bridge in PCIE SWITCH needs to be oriented to the above two application scenarios to complete the translation and address space sharing of various requests.

Disclosure of Invention

The technical problems to be solved by the invention are as follows: aiming at the problems in the prior art, the invention provides the PCIE exchanger with the optional functions of the transparent bridge and the non-transparent bridge and the multi-host system.

In order to solve the technical problems, the invention adopts the technical scheme that:

a PCIE exchanger with selectable transparent bridge and non-transparent bridge functions comprises a transparent bridge, a first selector, a second selector, a third selector and a non-transparent bridge, the fixed port of the first selector is used as the upstream port of the PCIE exchanger, the selection port of the first selector is respectively connected with the upstream port of the upstream PCI-PCI bridge of the transparent bridge and one selection port of the second selector, the select port of the second selector is coupled to a downstream port of a downstream PCI-PCI bridge of the transparent bridge, the fixed port of the second selector is divided into two paths, one path of the fixed port is connected with one selection port of the third selector after being connected with the non-transparent bridge in series, the other path of the fixed port is used as a bypass channel of the non-transparent bridge and is connected with the other selection port of the third selector, and the fixed port of the third selector is used as a downstream port of the PCIE exchanger.

Optionally, the non-transparent bridge includes an upstream endpoint device module UEP, a downstream endpoint device module DEP, a doorbell register module, and a scratchpad register module, and four paths are connected in parallel between the upstream endpoint device module UEP and the downstream endpoint device module DEP: the first path comprises an outbound address conversion module and an outbound request message ID conversion module; the second path comprises a message ID conversion module which responds to the outbound request and completes the message ID conversion; the third path comprises a message ID conversion module for responding the inbound request; the fourth path comprises an inbound address conversion module and an inbound request message ID conversion module; the upstream end point device module UEP is connected with the fixed port of the second selector in the upstream direction, the downstream direction is respectively connected with the outbound address conversion module, the outbound request response completion message ID conversion module, the inbound request message ID conversion module, the doorbell register module and the note register module, and the upstream end point device module UEP and the downstream end point device module DEP are standard PCIE end point devices.

Optionally, the upstream endpoint device module UEP has a plurality of base address registers, and is configured to allocate a configuration, an IO address, and a memory space in the first system connected to the upstream port of the PCIE switch; the downstream endpoint device module DEP is provided with a plurality of base address registers and is used for allocating configuration, IO addresses and memory spaces in a second system connected to an upstream port of the PCIE switch; the outbound address conversion module and the inbound address conversion module are used for realizing the address space conversion between the IO address and the memory space between the first system and the second system, the outbound request message ID conversion module and the inbound request message ID conversion module are used for realizing the request message ID conversion between the first system and the second system, and the outbound request response completion message ID conversion module and the inbound request response completion message ID conversion module are used for realizing the ID conversion between the outbound request response completion message and the inbound request response completion message between the first system and the second system.

Optionally, the detailed step of the outbound address translation module implementing address space translation of the IO address and the memory space between the first system and the second system includes: the outbound address conversion module carries out base address matching in a base address register on an address field of a received request message from a first system, if the address field of the received request message is matched with the base address of the first system, the offset of the message address field and the base address of the base address conversion register are recombined into an address in a second system address field, the base address in the second system field is stored in the base address conversion register, and if the address field of the received request message is not matched with the base address of the base address conversion register, the outbound address conversion module directly discards or indicates an upstream endpoint device module UEP to return to a state that the request is not supported; the detailed steps of the inbound address translation module for implementing address space translation between the IO address and the memory space of the first system and the second system include: the inbound address translation module matches the base address in the base address register for the address field of the received request message from the second system, if the address field matches the base address, the offset of the message address field and the base address of the base address translation register are recombined into the address in the first system address field, the base address in the first system field is stored in the base address translation register, and if the address field does not match the base address, the inbound address translation module directly discards or instructs the downstream endpoint device module DEP to return to the address field that does not support the request.

Optionally, the outbound request packet ID conversion module has a global table, the global table needs to be initialized after the first system enumeration configuration is completed and before the first system sends out the request, the table needs to design 8 entries, and each entry has 4 fields: the system comprises an INDEX field, a Bus field, a Dev field and a Func field, wherein the Func field is a function number of a first system domain, the INDEX field is a function number of a second system domain corresponding to the function number of the first system domain, the Bus field is used for recording a Bus number, the Dev field is used for recording a device number, and the lengths of the Dev field and the Dev field are 3 bits, 8 bits, 5 bits and 3 bits respectively; after receiving a first system domain request message ID = { FBus, FDev, FFunc }, an outbound request message ID conversion module, wherein the FBus represents a bus number, the FDev represents an equipment number, and the FFunc represents a function number; firstly, a global table is searched for by adopting a content matching search mode according to a request ID of a first system domain request message, FBus is used for matching a Bus field, FDev is used for matching a Dev field, FFunc is used for matching a Func field, and if any item in the global table is not hit, the request is returned to a state that the request is not supported; replacing the FFunc field of the request ID with the INDEX field of the hit table entry, so that the INDEX field of the hit table entry is a new FFunc field of the converted request ID of the second system request message, replacing the FDev field of the request ID with the device number DEPDev of the second system captured by the downstream endpoint device module DEP, and replacing the FBus field of the request ID with the device number DEPBus of the second system captured by the downstream endpoint device module DEP, so that the request message ID = { DEPBus, DEPDev, INDEX } converted to the second system domain is obtained.

Optionally, when the response inbound request completion packet ID conversion module realizes ID conversion of a response inbound request completion packet between the first system and the second system, the response inbound request completion packet ID = { DEPBus, depdevev, INDEX } from the second system domain is received, the INDEX field therein is used to access the global table of the outbound request packet ID conversion module, the Bus field, the Dev field, and the Func field of the hit table entry are read out, and the converted response inbound request completion packet ID = { Bus, Dev, Func } entering the first system is filled in the entry.

Optionally, the inbound request packet ID conversion module has a global table, the global table needs to be initialized after the first system enumeration configuration is completed and before the second system sends out the request, the table needs to design 8 entries, each entry has 4 fields: the system comprises an INDEX field, a Bus field, a Dev field and a Func field, wherein the Func field is a function number of a second system domain, the INDEX field is a function number of a first system domain corresponding to the function number of the second system domain, the Bus field is used for recording a Bus number, the Dev field is used for recording a device number, and the lengths of the Dev field and the Dev field are 3 bits, 8 bits, 5 bits and 3 bits respectively; after receiving a second system domain request message ID = { FBus, FDev, FFunc }, an outbound request message ID conversion module, wherein the FBus represents a bus number, the FDev represents an equipment number, and the FFunc represents a function number; firstly, a global table is searched for a request ID of a second system domain request message in a content matching search mode, FBus is used for matching a Bus field, FDev is used for matching a Dev field, FFunc is used for matching a Func field, and if any item in the global table is not hit, the request is returned to a state that the request is not supported; replacing the FFunc field of the request ID with the INDEX field of the hit table entry, so that the INDEX field of the hit table entry is a new FFunc field of the request ID of the converted first system request message, replacing the FDev field of the request ID with the device number DEPDev of the first system captured by the upstream endpoint device module UEP, and replacing the FBus field of the request ID with the device number DEPBus of the first system captured by the downstream endpoint device module DEP, so that the request message ID = { DEPBus, DEPDev, INDEX } converted to the first system domain is obtained.

Optionally, when the ID conversion module completes the ID conversion of the outbound request response completion packet between the first system and the second system, the inbound request response completion packet ID = { DEPBus, DEPDev, INDEX } from the first system domain is received, the INDEX field in the inbound request packet ID conversion module is used to access the global table of the inbound request packet ID conversion module, the Bus field, the Dev field, and the Func field of the hit entry are read out, and the converted inbound request response completion packet ID = { Bus, Dev, Func } entering the second system is filled in the incoming request response completion packet ID.

Optionally, the doorbell register module comprises a set of registers for passing interrupts between the first system and the second system, and the registers of the doorbell register module comprise: the system comprises a first system interrupt state register, a first system interrupt request register, a first system interrupt masking set register, a first system interrupt masking clear register, a second system interrupt state register, a second system interrupt request register, a second system interrupt masking set register and a second system interrupt masking clear register, wherein each register can be accessed by the first system and the second system through a memory address or IO address space, and effective interrupt is output as long as the request register has a position set and is not masked (mask); if the request bit is cleared or masked, the interrupt is invalidated; the note register module also comprises a group of registers, the first system and the second system can access the note register module through a memory address or IO address space and can be read and written, and the group of registers are used for transmitting control and state information or are used as read-write state registers.

In addition, the present invention also provides a multi-host system, which at least includes a first system and a second system, wherein the first system and the second system are connected by a PCIE switch, and the PCIE switch is a PCIE switch with the functions of the transparent bridge and the non-transparent bridge selectable.

Compared with the prior art, the invention has the following advantages: the invention comprises a transparent bridge, a first selector, a second selector, a third selector and a non-transparent bridge, wherein a fixed port of the first selector is used as an upstream port of a PCIE exchanger, a selection port of the first selector is respectively connected with an upstream port of an upstream PCI-PCI bridge of the transparent bridge and a selection port of the second selector, a selection port of the second selector is connected with a downstream port of a downstream PCI-PCI bridge of the transparent bridge, a fixed port of the second selector is divided into two paths, one path of the fixed port is connected with a selection port of the third selector after being connected with the non-transparent bridge in series, the other path of the fixed port is connected with the other selection port of the third selector as a bypass channel of the non-transparent bridge, and a fixed port of the third selector is used as a downstream port of the PCIE exchanger, the invention can realize that the functions of the transparent bridge and the non-transparent bridge are selectable, realize that an address route is compatible with an ID route, the method can be suitable for two typical application scenes of memory data exchange of two host systems and data exchange of a host system with a non-transparent port and equipment in another address domain to finish the conversion of various requests and address space sharing.

Drawings

Fig. 1 is a schematic structural diagram of a SWITCH system of a PCIE transparent bridge in the prior art.

Fig. 2 is a schematic structural diagram of a PCIE switch in the embodiment of the present invention.

Fig. 3 is a schematic diagram illustrating an implementation manner of address translation according to an embodiment of the present invention.

Fig. 4 is a schematic diagram illustrating an implementation manner of ID conversion in the embodiment of the present invention.

Detailed Description

As shown in fig. 2, the PCIE switch with selectable functions of a transparent bridge and a non-transparent bridge in this embodiment includes a transparent bridge, and further includes a first selector (in the figure, a selector 1), a second selector (in the figure, a selector 2), a third selector (in the figure, a selector 3), and a non-transparent bridge, where a fixed port of the first selector is used as an upstream port of the PCIE switch, a selection port of the first selector is respectively connected to an upstream port of an upstream PCI-PCI bridge of the transparent bridge and a selection port of the second selector, a selection port of the second selector is connected to a downstream port of a downstream PCI-PCI bridge of the transparent bridge, a fixed port of the second selector is divided into two paths, one path of the two paths is connected to a selection port of the third selector after being connected to the non-transparent bridge in series, and the other path is connected to another selection port of the third selector as a bypass channel of the non-transparent bridge, the fixed port of the third selector serves as a downstream port of the PCIE switch. Referring to fig. 2, in this embodiment, a selector 1, a selector 2, and a bypass channel (bypass path) are added between an Upstream (Upstream) port and a Downstream (Downstream) port of a conventional PCIE switch, and selection between a transparent bridge and a bypass channel is implemented through different configurations. A bypass channel is also added between the selector 2 and the selector 3, and a non-transparent bridge module is also added. The PCIE switch with selectable transparent bridge and non-transparent bridge functions in this embodiment may implement selection between the transparent bridge and the non-transparent bridge functions through different configurations. Selector 1 is coupled to the root complex or other PCIE switch in the upstream direction. In this embodiment, the host system in the upstream direction of the selector 1 is the first system. Selector 3 is coupled to the root complex or other PCIE switch in the downstream direction. I.e. the host system downstream of the selector 3 is the second system. It should be noted that: the definitions of upstream, downstream, Outbound (Outbound) and Inbound (Inbound) etc. that are described below, which indicate the direction of requests, are for the first system. For example, the Outbound (Outbound) direction is a request from a first system, the Inbound (Inbound) direction is a request from a second system, and so on. In addition, more selectors can be added between the downstream ports of other downstream PCI-PCI bridges and the upstream ports of the upstream PCI-PCI bridges of the transparent bridge as required to realize the selection of more paths, such as the selector 4 in the figure.

The root complex of the first system host is connected to the upstream port, selector 1 in fig. 2. The system architecture can work in 4 application scenarios. (1) The root complex of the second system host is connected to a non-transparent bridge port, selector 3 in fig. 2. If the application scenario using the SWITCH is that the first system and the second system are in equal roles, that is, they mainly exchange memory data, then the selector 1 and the selector 2 are configured to bypass the PCI-PCIE transparent bridges upstream and downstream, configure the selection signal of the selector 3, and select to output the non-transparent bridge to the downstream port, thereby implementing low-latency communication between the first system and the second system. (2) If the first system is a master and the second system is a slave, that is, the second system plays a backup role, the selection signals of the selector 1 and the selector 2 are configured, the PCI-PCIE transparent bridges at the upstream and the downstream are not bypassed, the selection signal of the selector 3 is configured, and the non-transparent bridge is selected to be output to the downstream port, so that the host of the second system can access the virtual bus inside the SWITCH, that is, the host can access other downstream endpoint devices on the virtual bus. (3) The non-transparent bridge module can be connected with other downstream ports by the realization or non-realization of the selector 4 and the control of corresponding selection signals, so that the configuration of the non-transparent ports is realized. (4) The SWITCH non-transparent bridge architecture proposed by the present invention is fully compatible with the conventional SWITCH architecture of fig. 1. Namely, the selector 1, the selector 2 and the selector 3 can be configured to bypass the non-transparent bridge, thereby realizing the expansion of compatibility.

As shown in fig. 2, the non-transparent bridge in this embodiment includes an upstream endpoint device module UEP (upstream UEP), a downstream endpoint device module DEP (downstream ep), a Doorbell (Doorbell) register module, and a note register module, where four paths are connected in parallel between the upstream endpoint device module UEP and the downstream endpoint device module DEP:

the first path includes an Outbound address translation module (abbreviated as Outbound address translation in fig. 2) and an Outbound request message ID translation module (abbreviated as ORID translation in fig. 2);

the second path includes a module for completing message ID conversion (abbreviated as OCID conversion in fig. 2) in response to the outbound request;

the third path includes a module for completing message ID conversion in response to the inbound request (abbreviated as ICID conversion in fig. 2);

the fourth path includes an Inbound address translation module (abbreviated as Inbound address translation in fig. 2), and an Inbound request message ID translation module (abbreviated as IRID translation in fig. 2);

the upstream direction of the upstream endpoint device module UEP is connected with the fixed port of the second selector, the downstream direction is respectively connected with the outbound address conversion module, the outbound request response completion message ID conversion module, the inbound request message ID conversion module, the doorbell register module and the note register module, and the upstream endpoint device module UEP and the downstream endpoint device module DEP are standard PCIE endpoint devices.

Referring to fig. 2, the upstream endpoint device module UEP has a plurality of base address registers (BARs for short) for allocating configuration, IO addresses and memory spaces in the first system connected to the upstream port of the PCIE switch; the downstream endpoint device module DEP is provided with a plurality of base address registers and is used for allocating configuration, IO addresses and memory spaces in a second system connected to an upstream port of the PCIE switch; the outbound address conversion module and the inbound address conversion module are used for realizing the address space conversion between the IO address and the memory space between the first system and the second system, the outbound request message ID conversion module and the inbound request message ID conversion module are used for realizing the request message ID conversion between the first system and the second system, and the outbound request response completion message ID conversion module and the inbound request response completion message ID conversion module are used for realizing the ID conversion between the outbound request response completion message and the inbound request response completion message between the first system and the second system.

Wherein the upstream endpoint device module UEP exposes the address space in the first system via 6 BARs in the UEP. The software of the first system allocates the configuration, IO and MEM spaces in the first system by configuring the type and size of the 6 BAR registers (base address registers) in the UEP. The upstream endpoint device module UEP is the end point of the PCIE topology in the first system, which is not visible in the second system.

The downstream end point equipment module DEP is connected with the selector 3 in the downstream direction, and the upstream direction is connected with the inbound address conversion module, the outbound request response completion message ID conversion module, the inbound request response completion message ID conversion module, the outbound request message ID conversion module, the doorbell register module and the note register module. The downstream endpoint device module DEP is a standard PCIE endpoint device and the address space exposed in the second system is implemented by 6 BARs in the downstream endpoint device module DEP. The software of the second system allocates the configuration, IO and MEM spaces in the second system by configuring the type and size of the 6 BAR registers in the downstream endpoint device module DEP. The downstream endpoint device module DEP is the end point of the PCIE topology in the second system, which is not visible in the first system.

The outbound address conversion module is connected with the upstream endpoint equipment module UEP and the outbound request message ID conversion module. The MEM read-write and the IO read-write are address routes, and address fields in the requests sent by the upstream endpoint device module UEP are converted by the outbound address conversion module, and information of the address fields is converted from the address space of the first system to the address space of the second system.

In this embodiment, the detailed step of the outbound address translation module implementing address space translation between the IO address and the memory space in the first system and the second system includes: the outbound address conversion module carries out base address matching in a base address register on an address field of a received request message from a first system, if the address field of the received request message is matched with the base address of the first system, the offset of the message address field and the base address of the base address conversion register are recombined into an address in a second system address field, the base address in the second system field is stored in the base address conversion register, and if the address field of the received request message is not matched with the base address of the base address conversion register, the outbound address conversion module directly discards or indicates an upstream endpoint device module UEP to return to a state that the request is not supported; the detailed steps of the inbound address translation module for implementing address space translation between the IO address and the memory space of the first system and the second system include: the inbound address translation module matches the base address in the base address register for the address field of the received request message from the second system, if the address field matches the base address, the offset of the message address field and the base address of the base address translation register are recombined into the address in the first system address field, the base address in the first system field is stored in the base address translation register, and if the address field does not match the base address, the inbound address translation module directly discards or instructs the downstream endpoint device module DEP to return to the address field that does not support the request.

The implementation principle of address translation is as shown in fig. 3, where a PCIE endpoint device Type0 configures the first 6 BARs for address allocation of various types of spaces required by the device by system software. With these 6 BARs, a device can apply for 6 32-bit spaces or 3 64-bit address spaces. It is assumed in fig. 3 that the white blank field in the first system address field is the spatial size of a certain BAR declaration of the upstream endpoint device module UEP and the white blank field in the second system address field is the spatial size of a certain BAR declaration of the downstream endpoint device module DEP. When a request from the first system hits the BAR of the upstream endpoint device module UEP, as shown by the hatched portion of the left diagonal line in fig. 3, the address of the request packet also has an address offset corresponding to the base address of the BAR. The function of the outbound address translation module is to translate the realm of the request to a corresponding address in the second system. The specific address translation is implemented as shown in fig. 3 right, the outbound address translation module performs BAR base address matching on the address field of the received request packet, and if matching is performed, the offset of the packet address field and the base address of the base address translation register are recombined into an address in the second system address field; if there is no match, the upstream endpoint device module UEP is directly discarded or instructed to return UR requests. Wherein the base address translation register stores the base address in the second system domain. The Inbound address translation module functions as the outbound address translation module and translates the requested address field in the second system to the corresponding address in the first system, except that the base address in the first system field is stored in the time base address translation register for Inbound address translation.

The outbound request message ID conversion module is connected with the outbound address conversion module and the downstream endpoint equipment module DEP. The outbound request message ID conversion module is connected with the outbound address conversion module and the downstream endpoint equipment module DEP. According to the definition of the PCIE protocol, in a header of the MEM read-write and IO read-write request message, in addition to an address field, there is ID field information of a request (request), where the ID is a Bus number (Bus), a device number (Dev), and a function number (Func) that are allocated to each node in the PCIE topology by the first system. Thus, in addition to address translation, MEM read and IO read and write requests also need to have their Requester ID translated to the request ID in the second system. The converted request is considered in the second system as a request initiated by the downstream endpoint device module DEP. Thus, the translated request ID is the topology node number of the downstream endpoint device module DEP in the second system.

In this embodiment, the outbound request packet ID conversion module has a global table, the global table needs to be initialized after the first system enumeration configuration is completed and before the first system sends out a request, the table needs to design 8 entries, each entry has 4 fields: the system comprises an INDEX field, a Bus field, a Dev field and a Func field, wherein the Func field is a function number of a first system domain, the INDEX field is a function number of a second system domain corresponding to the function number of the first system domain, the Bus field is used for recording a Bus number, the Dev field is used for recording a device number, and the lengths of the Dev field and the Dev field are 3 bits, 8 bits, 5 bits and 3 bits respectively; after receiving a first system domain request message ID = { FBus, FDev, FFunc }, an outbound request message ID conversion module, wherein the FBus represents a bus number, the FDev represents an equipment number, and the FFunc represents a function number; firstly, a global table is searched for by adopting a content matching search mode according to a request ID of a first system domain request message, FBus is used for matching a Bus field, FDev is used for matching a Dev field, FFunc is used for matching a Func field, and if any item in the global table is not hit, the request is returned to a state that the request is not supported; replacing the FFunc field of the request ID with the INDEX field of the hit table entry, so that the INDEX field of the hit table entry is a new FFunc field of the converted request ID of the second system request message, replacing the FDev field of the request ID with the device number DEPDev of the second system captured by the downstream endpoint device module DEP, and replacing the FBus field of the request ID with the device number DEPBus of the second system captured by the downstream endpoint device module DEP, so that the request message ID = { DEPBus, DEPDev, INDEX } converted to the second system domain is obtained.

The ID conversion is implemented according to the principle shown in fig. 4, and the ID conversion module of the Outbound request message has the function of converting the request ID field in the Outbound request message from the first system field to the ID value of the second system field. Note that the node number initiating the request is { FBus, FDev, FFunc }, that is, the request ID of the first system domain request message received by the outbound request message ID conversion module = { FBus, FDev, FFunc }. There is a global table entry in the outbound request packet ID conversion module, where the global table entry needs to be initialized after the first system enumeration configuration is completed and before the first system sends out a request, the table needs to design 8 entries, and each entry has 4 fields: an INDEX field, a Bus field, a Dev field, and a Func field, requiring 3 bits, 8 bits, 5 bits, and 3 bits, respectively. The reason for designing 8 items in the table is that the outbound request message ID conversion module only uses the FFunc field of the request ID of the first system domain request message when converting the request ID, and the function number field defined by the PCIE protocol has only 3 bits, so the global table only needs to design 8 items. In the conversion process, a global table is searched for by a CAM mode by the RequesterID of a first system domain request message, wherein the CAM mode is a content matching search mode, and if any { Bus, Dev and Func } in the table is not hit by { FBus, FDev and FFunc }, the request (UR request) is returned to be unsupported; if a certain item is hit, the FFunc field of the request ID of the first system domain request message is replaced by the INDEX field of the hit item, namely the INDEX field is the new FFunc field of the converted request ID of the second system request message. After the request message is converted, the second system host software looks that the request is initiated by the downstream endpoint device module DEP. Therefore, the FBus field and FDev field of the new request ID need to be replaced by the DEPBus value of the downstream endpoint device module and the DEPDev value of the downstream endpoint device module captured previously by the downstream endpoint device module DEP, that is, the request ID of the second system request packet = { the downstream endpoint device module DEPBus, the downstream endpoint device module DEPDev, INDEX }. This means that before the request is sent between the first system and the second system, the respective system host software has completed enumeration and configuration of the PCIE topology structure in the respective system, because the outbound request packet ID conversion module needs to use the downstream endpoint device module DEPBus value and the downstream endpoint device module DEPDev value captured by the downstream endpoint device module DEP, and the capture timing of each topology node is in the process of configuration after the respective host software enumerates the topology node, when the configuration request of each node arrives, the node records the bus field and the device field in the configuration request packet, and completes the capture action. After ORID translation, the request enters the second system, and the second system host software considers the originating node of the request as the downstream endpoint device module DEP. When the request reaches the destination node in the second system, the destination node returns a completion message.

The message ID conversion module is connected with the downstream end point equipment module DEP and the upstream end point equipment module UEP respectively in response to the inbound request. And the ICID module receives a completion message corresponding to the request message ID conversion module. The completion message defined by the PCIE protocol has a request ID field and a completion ID field, the former corresponds to the request ID field in the request message, and the latter corresponds to the topological node number of the completion message. Since the request is completed in the second system, the ICID needs to convert both the request ID field and the completion ID field into the topology node of the first system, that is, the request ID field is converted from the node number of the downstream endpoint device module DEP into the node number of the real request initiating endpoint in the first system, the completion ID field is converted from the completion node number in the second system into the node number of the upstream endpoint device module UEP in the first system, and the first system software perceives that the request is completed by the upstream endpoint device module UEP.

In this embodiment, when the message ID conversion module completes the ID conversion of the response inbound request completion message between the first system and the second system, the response inbound request completion message ID = { DEPBus, depdevev, INDEX } from the second system domain is received, the global table of the outbound request message ID conversion module is accessed using the INDEX field therein, the Bus field, the Dev field, and the Func field of the hit table entry are read out, and the converted response inbound request completion message ID = { Bus, Dev, Func } entering the first system is filled in the received response inbound request completion message ID. The completion packet is ID routed and has no address field, so when the completion packet passes through the downstream end point device module DEP and reaches the ID conversion module of the response inbound request completion packet, it is not necessary to perform address conversion on the completion packet. The first is to convert the request ID of the second system to the original ID of the originating node of the request in the first system. In the completion message, the request ID = { downstream endpoint device module DEPBus, downstream endpoint device module DEPDev, INDEX }, then the global table needs to be indexed by using a lookup table manner, that is, the global table is accessed by using INDEX as an address item, the Bus number, device number and function number field of the hit table item are read out, and the converted completion message entering the first system is filled, that is, the request ID = { Bus, Dev, Func } at this time. Because the global table hit by the CAM lookup method used in the ID conversion performed by the ORID module means that { FBus, FDev, FFunc } = { Bus, Dev, Func }, the response inbound request completion packet ID conversion module is correct for the request ID field in the completion packet. The completion ID field in the completion packet does not need to look up a global table, and only needs to replace each field with a field captured by the upstream endpoint device module UEP. This is reasonable because in the view of the host software of the first system, the second system address field is not visible, and all requests addressed to the second system address field are considered by the host software of the first system to be addressed to the upstream endpoint device module UEP, so the node IDs allocated by the upstream endpoint device module UEP in the PCIE topology of the first system should be filled in the completion ID fields of all completion packets.

The inbound address conversion module is connected with the downstream endpoint equipment module DEP and the inbound request response completion message ID conversion module. The module converts address fields in MEM read-write and IO read-write requests sent by a downstream endpoint device module DEP, and converts information of the address fields from an address space of a second system to an address space of a first system. Likewise, such Inbound requests from the second system require ID translation in addition to address translation. ID conversion is implemented in an ID conversion module that completes a message in response to an inbound request. And the response inbound request completion message ID conversion module is connected with the inbound address conversion module and the upstream endpoint equipment module UEP. The response inbound request completion message ID conversion module converts the request ID of the inbound request to the request ID in the first system. The converted request is considered to be a request initiated by the upstream endpoint device module UEP in the first system. Therefore, the converted request ID is the topology node number of the upstream endpoint device module UEP in the first system.

In this embodiment, the inbound request packet ID conversion module has a global table, the global table needs to be initialized after the first system enumerates and configures, and before the second system sends out a request, the table needs to design 8 entries, each entry has 4 fields: the system comprises an INDEX field, a Bus field, a Dev field and a Func field, wherein the Func field is a function number of a second system domain, the INDEX field is a function number of a first system domain corresponding to the function number of the second system domain, the Bus field is used for recording a Bus number, the Dev field is used for recording a device number, and the lengths of the Dev field and the Dev field are 3 bits, 8 bits, 5 bits and 3 bits respectively; after receiving a second system domain request message ID = { FBus, FDev, FFunc }, an outbound request message ID conversion module, wherein the FBus represents a bus number, the FDev represents an equipment number, and the FFunc represents a function number; firstly, a global table is searched for a request ID of a second system domain request message in a content matching search mode, FBus is used for matching a Bus field, FDev is used for matching a Dev field, FFunc is used for matching a Func field, and if any item in the global table is not hit, the request is returned to a state that the request is not supported; replacing the FFunc field of the request ID with the INDEX field of the hit table entry, so that the INDEX field of the hit table entry is a new FFunc field of the request ID of the converted first system request message, replacing the FDev field of the request ID with the device number DEPDev of the first system captured by the upstream endpoint device module UEP, and replacing the FBus field of the request ID with the device number DEPBus of the first system captured by the downstream endpoint device module DEP, so that the request message ID = { DEPBus, DEPDev, INDEX } converted to the first system domain is obtained.

And the ID conversion module for completing the message in response to the outbound request is respectively connected with the DEP and the UEP. The ID conversion module initiates a request message in the first system in response to the inbound request completion message, and receives a completion message in response to the outbound request completion message. Because the request is completed in the first system, the ID conversion module for completing the message in response to the outbound request needs to simultaneously convert the request ID field and the completion ID field into the topology node of the second system, that is, the request ID field is converted from the node number of the upstream endpoint device module UEP to the node number of the real request initiating endpoint in the second system, the completion ID field is converted from the completion node number in the first system to the node number of the downstream endpoint device module DEP in the second system, and the second system software perceives that the request is completed by the downstream endpoint device module DEP.

In this embodiment, when the ID conversion module completes the ID conversion of the outbound request response completion packet between the first system and the second system, the inbound request response completion packet ID = { DEPBus, depdevev, INDEX } from the first system domain is received, the INDEX field therein is used to access the global table of the inbound request packet ID conversion module, the Bus field, the Dev field, and the Func field of the hit table entry are read out, and the converted inbound request response completion packet ID = { Bus, Dev, Func } entering the second system is filled in the received inbound request response completion packet ID.

As can be seen from the foregoing, the message translation process for outbound requests and inbound requests is reversed in direction, but functions in unison. The message conversion process of the inbound completion message and the outbound completion message are in the same direction, but the functions are the same.

The doorbell register module and the memo register module are respectively connected with the downstream endpoint device module DEP and the upstream endpoint device module UEP, wherein the doorbell register is used for transmitting an interrupt; the note register is used to pass information such as status.

In this embodiment, the doorbell register module comprises a set of registers (each register has 16 bits in this embodiment) for transferring interrupts between the first system and the second system, and the registers of the doorbell register module include: the system comprises a first system interrupt state register, a first system interrupt request register, a first system interrupt masking set register, a first system interrupt masking clear register, a second system interrupt state register, a second system interrupt request register, a second system interrupt masking set register and a second system interrupt masking clear register, wherein each register can be accessed by the first system and the second system through a memory address or IO address space, and as long as the request register has a position set and is not masked, a valid interrupt is output; if the request bit is cleared or masked, the interrupt is invalidated (deassert); in the embodiment, in the handover state, after the jump between the interrupt valid (assert) and the interrupt invalid (deassert), corresponding messages need to be further transferred upwards, for example, an interrupt message INTx, a message notification interrupt message MSI, and the like.

In this embodiment, the memo register module also includes a set of registers (usually 8 registers), which can be accessed by both the first system and the second system through a memory address or an IO address space, and are readable and writable, and the set of registers is used for transferring control and status information or simply used as a read-write status register. Reading and writing the set of registers without poking out an interrupt.

Other types of requests, such as configuration read and write requests and messages, are defined by the PCIE protocol. The configuration read-write request does not pass through the non-transparent bridge, and the configuration read-write request is finished when reaching the non-transparent bridge, because the configuration request of the first system only hits the upstream endpoint device module UEP, and the configuration request of the second system only hits the downstream endpoint device module DEP. The message type request cannot pass through the non-transparent bridge, the message is directly discarded after reaching the non-transparent bridge, and then the communication between the first system and the second system is completed through the doorbell register module and the note register module.

In summary, the PCIE switch with the functions of the transparent bridge and the non-transparent bridge selectable in the present invention can implement the functions of the transparent bridge and the non-transparent bridge selectable, implement address routing and ID routing compatibility, and can adapt to two typical application scenarios, that is, two host systems exchange memory data, and a host system of a non-transparent port exchanges data with a device of another address domain, to complete conversion of various requests and address space sharing.

In addition, the present invention also provides a multi-host system, which at least includes a first system and a second system, wherein the first system and the second system are connected by a PCIE switch, and the PCIE switch is a PCIE switch with the functions of the transparent bridge and the non-transparent bridge selectable.

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.