阅读说明：本技术 通信链路重新训练 (Communication link retraining ) 是由 B·麦克劳克林 于 2020-09-25 设计创作，主要内容包括：本文描述的示例涉及确定设备是否能够重新训练另一设备的一个或多个组件的设置。一些示例包括：通过以下操作进行链路重新训练：由第一设备中的接收机通过通道从第二设备中的发射机接收信号,该信号包括识别重新训练链路的能力的第一通信；从第一设备发送第二通信,其包括能力待调整的第二设备的一个或多个组件和对修改一个或多个组件的一个或多个参数的请求；以及在第一设备处接收识别重新训练的状态的第三通信。在一些示例中,一个或多个组件包括均衡器,并且一个或多个参数包括至少一个抽头设置。在一些示例中,一个或多个参数包括前体、主体或后体均衡设置。(Examples described herein relate to determining whether a device is capable of retraining settings of one or more components of another device. Some examples include: link retraining is performed by: receiving, by a receiver in a first device, a signal from a transmitter in a second device over a channel, the signal including a first communication identifying a capability to retrain a link; sending, from the first device, a second communication comprising one or more components of the second device whose capabilities are to be adjusted and a request to modify one or more parameters of the one or more components; and receiving, at the first device, a third communication identifying a retrained state. In some examples, the one or more components include an equalizer and the one or more parameters include at least one tap setting. In some examples, the one or more parameters include a precursor, subject, or afterbody equalization setting.)

1. An apparatus, comprising:

a first device associated with a lane of a communication link, wherein:

the first device comprises a transmitter and a receiver;

the first device is to receive a first communication identifying a capability to retrain a link;

the first device is to send a second communication identifying one or more components, and the second communication is to cause one or more parameters of the one or more components to be modified; and is

The first device is to receive a third communication identifying a retraining state.

2. The apparatus of claim 1, wherein the one or more components comprise an equalizer and the one or more parameters comprise at least one tap setting.

3. The apparatus of claim 1, wherein the one or more parameters comprise a precursor, subject, or postcursor equalizer setting.

4. The apparatus of claim 1, wherein the first device is to:

advertising one or more capabilities;

receiving one or more capabilities from the second device; and

based on the advertised and received capabilities, an operating mode for the channel is selected.

5. The apparatus of claim 4, wherein the one or more capabilities comprise one or more of a link speed, a Forward Error Correction (FEC) capability, or a pause capability.

6. The apparatus of claim 1, wherein:

the first communication includes a setup message identifying at least one tap that is adjustable through link retraining and including a local port identifier of a local port to be the subject of a tap change request by the first device.

7. The apparatus of claim 1, wherein the second communication comprises a request message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, changing a tap setting, or maintaining a tap setting.

8. The apparatus of claim 1, wherein the third communication comprises a response message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, an updated indication, an un-updated indication, an indication of reaching a minimum value, or an indication of reaching a maximum value.

9. The apparatus of claim 1, wherein the first device is to:

link retraining is initiated in response to a change in power, voltage, or temperature, or elapsed time.

10. The apparatus of claim 1, comprising one or more of: a server, a rack, or a data center, and wherein the first device is disposed in one or more of: a server, a rack, or a data center.

11. A method for link retraining, the method comprising:

receiving, by a receiver in a first device, a signal from a transmitter in a second device over a channel, the signal including a first communication identifying a capability to retrain a link;

sending a second communication from the first device, the second communication comprising one or more components of a second device whose capabilities are to be adjusted and a request to modify one or more parameters of the one or more components; and

receiving, at the first device, a third communication identifying a retrained state.

12. The method of claim 11, wherein the one or more components comprise an equalizer and the one or more parameters comprise at least one tap setting.

13. The method of claim 11, wherein the one or more parameters comprise a precursor, body, or afterbody equalization setting.

14. The method of claim 11, wherein the first communication comprises a setup message identifying at least one tap that is adjustable through link retraining and comprising a local port identifier of a local port to be the subject of a tap change request by the first device.

15. The method of claim 11, wherein the second communication comprises a request message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, changing a tap setting, or maintaining a tap setting.

16. The method of claim 11, wherein the third communication comprises a response message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, an updated indication, an un-updated indication, an indication of reaching a minimum value, or an indication of reaching a maximum value.

17. The method of claim 11, comprising:

one lane at a time is trained whereby the first device makes a tap request and adjusts its receiver, and then the link partner of the first device begins link retraining for its receiver.

18. The method of claim 11, comprising:

link retraining is initiated in response to a change in power, voltage, or temperature, or elapsed time.

19. A computer-readable medium having instructions stored thereon that, if executed by one or more processors, cause the one or more processors to:

link retraining is performed by:

receiving, by a receiver in a first device, a signal from a transmitter in a second device over a channel, the signal including a first communication identifying a capability to retrain a link;

sending a second communication from the first device, the second communication comprising one or more components whose capabilities are to be adjusted and a request to modify one or more parameters of the one or more components; and

receiving, at the first device, a third communication identifying a retrained state.

20. The computer-readable medium of claim 19, wherein the one or more components comprise an equalizer and the one or more parameters comprise at least one tap setting.

21. The computer-readable medium of claim 19, wherein the first communication comprises a setup message identifying at least one tap that is adjustable through link retraining and including a local port identifier of a local port to be the subject of a tap change request by the first device.

22. The computer-readable medium of claim 19, wherein the second communication comprises a request message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, changing a tap setting, or maintaining a tap setting.

Background

The optical module is a hot-pluggable optical transceiver for use in high bandwidth data communication applications. Optical modules typically have electrical connections to a chip, such as a host Application Specific Integrated Circuit (ASIC). The electrical connection is a high data rate serial link (50GAUI-1) with a current rate of 50Gb/s per lane (referred to as xGAUI-n, where AUI stands for attached cell interface, x is the data rate per lane in gigabits per second (Gbps), and n is the number of lanes). 50GAUI-1 is a single channel serial link for chip-to-chip or chip-to-module interconnection. However, the optical module is only one example of a device that uses a serial link. The electrical signal receiving and processing module may also use a serial link.

Serial communication between a transmitter and a receiver at very high data rates uses equalization to mitigate frequency dependent signal attenuation. Equalization may be applied at the transmitter (Tx) and/or at the receiver (Rx). Typically, a combination of transmitter and receiver equalization is used.

Drawings

FIG. 1 depicts an example system.

Fig. 2A-2B depict example systems.

FIG. 3A depicts example contents of a field of a next page.

Fig. 3B illustrates an example second next page format.

Fig. 4A depicts a process sequence for link training.

Fig. 4B depicts an example sequence that may be used for link retraining operations.

Fig. 5A depicts an example of an equalizer.

Fig. 5B depicts a functional model of the structure of a four-tap Feed Forward Equalizer (FFE).

Fig. 6A and 6B depict an example process.

Fig. 7A and 7B depict an example process.

Fig. 8 depicts a network interface.

Fig. 9 depicts an example switch.

FIG. 10 depicts an example system.

FIG. 11 depicts an environment.

Detailed Description

Common communication capabilities and parameters are discovered among network connected devices for devices communicating with each other. Auto-negotiation (AN) is a process by which the endpoints of a link share information about various capabilities related to their communication. For AN example, see clause 73 of the Institute of Electrical and Electronics Engineers (IEEE) 802.3-2018. The link partner device swaps capabilities and modes of operation via the swap base page, and if requested, the link partner device swaps the next page. According to clause 73 of IEEE802.3-2018, each device sends a list of its data rate capabilities to its link partner. Auto-negotiation may determine the highest common capability, and the highest common capability is used for communication between link partner devices. After both devices receive the capability list of their link partners, the devices may transition to the highest common data rate and feature capabilities.

Link training is a process used by devices connected through copper cables, backplanes, or other wired or wireless signal transmission media by which transmitters and receivers on high-speed serial links communicate with each other to tune their equalization settings. For example, a serializer/deserializer (SerDes) may use link training. Link training enables tuning a Finite Impulse Response (FIR) filter for each channel in an Application Specific Integrated Circuit (ASIC) or other device to achieve a desired Bit Error Rate (BER), eye size, signal-to-noise ratio (SNR), or link error (e.g., uncorrectable and correctable Forward Error Correction (FEC) error, pseudo-random bit sequence (PRBS) error, Physical Coding Sublayer (PCS) error). In some examples, the receiver examines the eye pattern after applying equalization to the signal and determines whether the eye pattern height and/or eye pattern width is acceptable. The receiver may make a decision to terminate link training because the eye pattern is acceptable or may remain trained to further optimize the eye pattern. The eye check process begins again if the receiver requests its link partner transmitter to change the front body (presursor), main body (main radar), or post-body (post-radar) equalization settings.

Since the link partners both include a transmitter and a receiver, the link partner can simultaneously train the transmitter of another partner. After training the link, both devices begin transmitting normal data traffic using the optimized transmitter settings.

The ethernet over backplane and copper (IEEE 802.3) standard for 10Gb/s and above includes a PMD (physical medium dependent) control function that enables the adjustment of transmitter equalization settings as part of link training. The PMD control function uses a handshake-based protocol for requesting coefficient changes. The protocol is described by a state diagram (e.g., FIGS. 72-4, 72-5, and 72-6 in IEEE Std 802.3-2012, and variations thereof). These state diagrams are referenced in approved and draft standards for multiple PMDs (e.g., 10GBASE-KR, 40GBASE-KR4, 40GBASE-CR4, and 100GBASE-CR 10).

IEEE802.3-2018 clause 73 and subclauses 73.7.4 and 73.7.5 and tables 73-7 state that the device under test will delay the correct amount of time before attempting to verify the link state as determined by the auto-negotiation process. IEEE802.3-2018 clauses 72 and 73 specify appropriate link training times. If the link _ fail _ inhibit _ timer has expired before the link is active (e.g., the signal is decoded correctly), the link fails. However, in some cases, the amount of time set by the link _ fail _ inhibit _ timer may not be sufficient to achieve a desired Bit Error Rate (BER) on the link, resulting in more errors during normal operation (e.g., more Cyclic Redundancy Check (CRC) errors).

Reference herein to any standard refers to any version (including previous, current and future versions) and proprietary derivatives thereof.

Various embodiments attempt to improve link quality using auto-negotiation, at least in cases where IEEE-defined link training times may not allow for optimal training. IEEE802.3 proposes negotiation using the next page exchange phase. See appendix 28C of IEEE802.3-2018 for an example of a next page message. In some examples, the next page may be used to exchange identifier tags, Energy Efficient Ethernet (EEE) parameters, operating parameters, and vendor specific information. According to various embodiments, one or both link partners may use a next page exchange during auto-negotiation to advertise a capability to extend the link training time and an amount of time that the link partner may extend the link training time. In some examples, the link partner may advertise one or more amounts of time that the link training time can be extended. The link partner may negotiate how much to extend the "link _ fail _ inhibit _ timer" to set the amount of time for link training. In some examples, if both parties advertise the capability to do so, the highest common denominator of the two extension times is used and added to the link failure prohibit time to determine the total amount of link failure prohibit time allowed. However, if the link partner indicates that extension of the link failure prohibit time is not supported, then a default prohibit time (e.g., IEEE802.3 default link _ fail _ inhibit _ timer) is used.

Various embodiments may be used for chip-to-chip link training or link retraining over copper cables (e.g., 40GBASE-CR4 or derivatives thereof) on backplane connections (e.g., 10GBASE-KRx or derivatives thereof (where x is an integer)) or network interface to network interface connection traces. Various embodiments may be applied to link or channel speeds at 10Gbps or higher or any link or channel speed.

Various embodiments provide a way for partner or port agreements to extend IEEE-defined link training times, allowing for more optimal equalizer tuning. Receipt of the proprietary "next page" may represent, among other reasons, a link partner's request to extend a subsequent link training phase for a specified time. The maximum amount of time allowed for extended link training may be specified in the field of the next page. For example, the value in the field represents additional time to allow beyond the maximum time defined by the IEEE.

Many SerDes have difficulty achieving good Rx equalization (e.g., eye diagram quality, signal integrity) in the allotted time under IEEE 802.3. Extending the link failure prohibit time may be used for more time for link training to improve the link. By extending the link training time, the Bit Error Rate (BER) on the link may be minimized, resulting in fewer errors during normal operation (e.g., fewer CRC errors).

Link training is applicable to other wired communication or networking systems such as, but not limited to, FibreChannel, InfiniBand, or serial attached small computer system interface (SAS). Extending link training time may be useful for 4-level Pulse Amplitude Modulation (PAM) links (e.g., PAM4 links), PAM4, PAM5, PAM6, n-level PAM links (where n is an integer), non-return-to-zero (NRZ) line codes, and so forth.

Various embodiments provide a protocol that may be invoked at any time after a point-to-point link is established. Various embodiments provide at least a mechanism to allow tuning to continue after the link is established. Various embodiments may be used to extend the tuning of the link beyond the time allowed by IEEE802.3 for link training after auto-negotiation. In some cases, a protocol may be used after establishing a link to modify transmit (Tx) equalization on both sides of the link. Some embodiments provide for requesting a link partner to make changes to the link partner's transmit settings to optimize the local receiver's capabilities. Link tuning may be desirable for various reasons, such as changed conditions (e.g., power, voltage, temperature changes), periodic changes, and other reasons. Various embodiments may provide for extended link training after a link is completed, whether done using AN or not. Extended link training may allow the tuning link partner to send equalization settings as well as local receiver equalization, resulting in potentially improved tuning. Link training may occur from the perspective of the receiver so the transmitter settings are requested to be changed. In some examples, data may be sent while training occurs, and training data patterns may be used during retraining. During link retraining, a transition density signal or different separate training data may be used.

Various embodiments may use a Link Layer Discovery Protocol (LLDP) type-length-value (TLV) to request incremental changes to the link partner's transmit equalization settings (e.g., front, master, back). However, the LLDP protocol and TLV format are not required and any type of message may be used. For example, a packet header may be used to convey transmitter or receiver equalizer settings. For example, various embodiments may use User Datagram Protocol (UDP) packet switching to enable and use extended link training. The extended link training, in combination with the quality metric for local receiver equalization, may be used to improve link quality (e.g., lower Bit Error Rate (BER)) and reduce identified link errors at the receiver (e.g., Forward Error Correction (FEC) or PCS) over receiver-only optimization. Extended link training or retraining may be used to extend the tuning of the link beyond the time allowed by IEEE802.3 (or other standards or specifications) for link training.

According to some embodiments, the initiator device may use LLDP compatible messages to communicate or request the partner device to determine or check whether the received signal characteristics have drifted or whether the signal quality is acceptable, and to trigger either the initiator device or the partner device to perform retraining. For example, according to some embodiments, a temperature change (e.g., hot during the day, or cold during the night or day) at a base station or edge or fog computing device or any type of device may trigger retraining of the equalizer settings.

For example, a base station supporting communication using wired or wireless protocols (e.g., 3GPP Long Term Evolution (LTE) (4G) or 3GPP 5G), indoor (on-premise) data centers, outdoor (off-premise) data centers, edge network elements (computing elements provided physically closer to a base station or network access point than the data center), fog network elements (computing elements provided physically closer to a base station or network access point than the data center but further from the edge network), and/or hybrid data centers (e.g., data centers that use virtualization, cloud, and software-defined networking to deliver application workloads across the physical data center and in a distributed, cloudy environment) may apply link training time extension and retraining. The network or computing element may be used in a Local Area Network (LAN), Metropolitan Area Network (MAN), network with devices connected using fiber optic links, Campus Area Network (CAN), or Wide Area Network (WAN).

Various embodiments may be used for 50G SerDes speeds or higher, although lower speeds may be supported.

Fig. 1 is a block diagram showing an ethernet port circuit in the network interface controller 50. The ethernet port logic includes a Media Access Control (MAC) module 52, a coordination sublayer module 54, and a PHY module 56. The PHY module 56 may include a Physical Medium Attachment (PMA) sublayer module 62, a Physical Medium Dependent (PMD) sublayer 64, a Forward Error Correction (FEC) module 60, and a Physical Coding Sublayer (PCS) module 58.

The MAC module 52 is configured to transfer data in and out of the PHY module 56. The coordination sublayer (RS) module 54 may provide a mapping operation that coordinates signaling at the Medium Independent Interface (MII) with the Medium Access Control (MAC) -physical signaling sublayer (PLS) service definition. The MAC module 52 may be configured to implement aspects of the MAC layer operations and the RS module 54 may be configured to implement the coordination sublayer operations.

The Physical Medium Dependent (PMD) sublayer 64 may be responsible for interfacing to the transmission medium, Medium Dependent Interface (MDI) 80. The Physical Medium Attachment (PMA) sublayer 62 may perform transmit, receive, signal detection, clock recovery, and skew alignment. The PMD 64 and PMA 62 may be configured to send and receive serial data through the MDI 80.

In some examples, the PMD 64, PMA 62a and/or 62b may include or use a SerDes. In some examples, extended link training and retraining may be provided to adjust filter parameters of transmit and/or receive equalizers used by the SerDes. For example, a software SerDes driver executed by a processor in a host or network interface may be used to change transmit equalizer parameters. In some examples, any combination of hardware, software, and/or firmware may be used to manage and perform link training and/or link retraining.

In some examples (e.g., for 100GBASE-CR1 or 100GBASE-KR1), FEC module 60 may decode data passed from PMD 64 and PMA 62 to PCS module 58, or encode data passed from PCS module 58 to PMD 64 and PMAs 62a, 62 b. In some examples, (e.g., for the 200G and 400G modes), the PCS module 58 includes the FEC module 60. Forward error correction codes may improve the reliability of data transmission at higher line speeds.

In the transmit direction, the MAC module 52 may receive data to be transmitted in a Medium Access Control (MAC) frame through the MDI 80 and generate a MAC frame including an inter-packet gap (IPG), a preamble, a start-of-frame delimiter (SFD), a padding, and a Cyclic Redundancy Check (CRC) bit in addition to the received data, and then pass the MAC frame to the PHY module 56. PHY module 56 may encode the MAC frame for reliable serial transmission through MDI 80.

In the receive direction, the MAC module 52 may receive MAC frames from the PHY module 56 over the data bus. The MAC module 52 may accept MAC frames from the PHY 56, perform ethernet frame detection and verification, Cyclic Redundancy Check (CRC) verification, update statistics counters, remove CRCs, preamble detection and removal, and Start of Frame Delimiter (SFD) detection and removal, and forward the rest of the MAC frame including headers for other protocols to the next layer (e.g., the Internet Protocol (IP) layer) for processing.

Fig. 2A shows a simplified example for a transmitter-receiver pair between the network interface controller 100 and the device 120. The MDI 130 provides a link between the network interface controller 100 and the device 120 by transmitting data in parallel on one or more lanes. Device 120 may be any device (e.g., another network interface, a Network Interface Card (NIC), a switch, a router, a server, a host computing platform, etc.).

The network interface controller 100 may include a host receiver 206 and a host transmitter 208 for at least one channel of an electrical link between the network interface controller 100 and the device 120. The device 120 may include a module receiver 212 and a module transmitter 210 for an electrical link between the network interface controller 100 and the device 120.

For example, link training controller 202 of NIC100 may be configured to advertise and negotiate with the link training controller of module 120 for an extension of the link training time, and vice versa. Link training controller 202 may also initiate or manage link retraining operations, as described herein. The link training controller may be implemented as a driver, microcontroller, or other software in the host or network interface.

The transmitter (Tx)208/210 or receiver (Rx)206/212 may use SerDes to serialize or deserialize a signal. Rx tuning may be used to restore (clear-up) signal quality when the SerDes is turned on and signals are received. When there is a time limit to perform Rx tuning, the signal is to pass to the PCS layer within the time limit and the link appears if it is acceptable. If the link fails, training may be resumed. In some examples, Tx 208-Rx 212 and/or Tx 210-Rx 206 may utilize independent Rx tuning. In some embodiments, the amount of time to perform equalizer tuning is the same for Tx 208-Rx 212 and Tx 210-Rx 206.

When auto-negotiation is used to establish a link between two ethernet ports, IEEE defined procedures are followed. First, a "BASE page" swap may be performed to determine common capabilities and select an operating mode (e.g., link speed (e.g., 1000BASE-KX, 10GBASE-KX4 … … 100GBASE-CR4, etc.), FEC mode, pause capabilities, etc.). Next, the next page exchange stage of arbitrary length can be performed. The next page swap may be used to advertise IEEE capabilities as well as non-IEEE capabilities (e.g., ethernet alliance mode). At the end of the next page swap, the selected mode of operation may be configured and the link training phase may begin. During this link training phase, changes to the peer-to-peer transmit (e.g., Tx 208 or Tx 210) equalization settings, monitoring the impact on link quality at the receiver (e.g., Rx 206 or Rx 212) and adjusting the equalization settings to optimize the link may be made.

In some examples, the link training time may be extended by specifying an earlier start time and using a default link training time. For example, the device may negotiate a start time for link training by negotiating AN offset from a default start of a link training time (e.g., using AN, next page swap, or proprietary swap scheme), where the offset indicates AN amount of time before the default start of the link training time to start link training. The device may indicate a capability to begin link training prior to a default start of a link training time, indicate a maximum amount of time prior to the default start of the link training time to begin link training, and select the lesser of the amounts of time prior to the default start of the link training time to begin link training.

Communication between devices may occur using any protocol. For example, an ethernet frame may be sent by the NIC100 to the device 120. For example, an ethernet frame may be sent by the device 120 to the NIC 100. The ethernet frame may include one or more of the following: a preamble, a Start of Frame Delimiter (SFD), a destination MAC address, a source MAC address, an EtherType field, a length field, a frame check sequence (e.g., Cyclic Redundancy Check (CRC)), and a payload.

From the perspective of either port, there are four methods for link training, but more or fewer methods may be used.

(1) The local port neither sends the proprietary next page nor receives it. This results in no extension of the link training phase.

(2) The local port sends the private next page but does not receive it from the link partner (in response to which it will receive a NULL page). This indicates that the link partner is either unable or unwilling to extend the link training time. This results in no extension of the link training phase.

(3) The local port is unable or unwilling to extend the link training time and therefore does not send the exclusive next page, but receives it from the link partner (it will respond with a NULL page). This results in no extension of the link training phase.

(4) The local port both sends and receives a private next page indicating that both ports are able and willing to extend the link training time. The amount of time to extend the link training phase is defined as the minimum of the time advertised by the two ports. In case (4), the subsequent link training phase will be allowed to last longer (if necessary) than the maximum time defined by the IEEE. If the time is still exceeded, the auto-negotiation will be restarted.

According to various embodiments, the link training controller advertises extended link training capabilities during the next page exchange phase of auto-negotiation. There are two next pages required to advertise this capability. First, the first next page formatted with an Organization Unique Identifier (OUI) tag is sent with the Vendor OUI of < tbd > using message code # 1. Next, the second next page is unformatted by sending the OUI tag with an extension of the requested time in milliseconds. An example format for the first next page is shown in fig. 3A.

In fig. 3A, the following is example content of the fields of the next page.

Fig. 3B illustrates that an example second next page format may be as follows. Example contents of the fields may be as follows.

D0_3:0001b (indicating that an "extended Link training time" value follows).

D4_8:00000b

D9_10 vector OUI < tbd > bits [1:0]

D11:T

D12_13:00b

D14:ACK

D15:NP

D16 — 47 time in milliseconds (unsigned 32 bit value) for extension with respect to link training. In some examples, the value is an absolute time for link training, which may be less than, equal to, or greater than an IEEE802.3 default link training time.

In some examples, if one side of the link does not support "provider link training extension capability," it will respond with a NULL page to the "next page" formatted in OUI tags. This may have the effect: the value of "0" is announced for the "Time _ in _ ms" field.

In some examples, if one side of the link does not support the option "extended link training time," it will respond with a NULL page to the next page unformatted in OUI tags. This may have the effect: the value of "0" is announced for the "Time _ in _ ms" field.

The time allowed for link training resolves to the highest common denominator of the values advertised by the two parties to the link. If the parsed time is less than the IEEE (or Ethernet alliance) defined link training time, then the IEEE (or Ethernet alliance) defined link training time is used instead.

The link training time specified in the next page may be: (1) a value (e.g., a relative value) to be added to the default IEEE link training time, or (2) an absolute link training time to be used. In (1), there is no possibility that the negotiated link training time may be less than the default IEEE link training time, and in (2), the link training may be less than the default IEEE link training time.

In some examples, the default link training time may be set to a higher or lower value to extend or shorten the link training time. For example, AN, next page swap, or proprietary swap schemes may be used to set or change the default link training time. In some examples, the extension or shortening of the link training time may or may not be used with the default link training time set or changed.

Referring next to fig. 2B, in some embodiments, after link training is complete, the microcontroller 244 (e.g., any one of 244-0 through 244-N) associated with any channel may initiate retraining. For example, a device driver, platform device, or software may trigger negotiation and implementation of link retraining. Link retraining may be performed independently per SerDes lane. Various embodiments provide a protocol to establish retraining, provide requests related to retraining, or receive responses related to retraining. A protocol stack (e.g., layer 3) may recognize a message as indicating a setup, request, or response.

In some examples, various messages may be used to initiate (establish) link training to indicate supported taps, requesting application of specific equalizer tap parameters (e.g., hold, increment, or decrement Tx tap # x). In some examples, instead of incrementing or decrementing, one or more increases or one or more decreases may be applied. Various embodiments provide a response that indicates when a particular equalizer tap parameter is used (e.g., Tx tap # x has been updated). Tx taps may be defined using signed integer values (-3..0. + 3). The pre-emphasis taps may be identified by negative values, while the post-emphasis taps may be identified by positive values. The main tap can be identified by the value zero (0).

In some cases, not all taps may be retrained by a given SerDes or its controller. In some examples, the transmitter may advertise any taps that can be adjusted by link retraining. The supported taps may be communicated to the link partner in an initial setup message. The setup message may identify the local Port _ id that will be the subject of the tap change request by the link partner. Port _ id may be provided in any response. In some examples, the Port _ id value may be any 8-bit unsigned value (or any other value (e.g., 32 bits)) that the local device may use to map to a local Port. Since the link occurs before the protocol is used, both parties are considered to agree on the number and mapping of SerDes lanes used by the port.

The supported taps may be identified by 8-bit unsigned values containing a mapping of the supported Tx taps. For example, the mapping from bit to tap number is as follows:

tap number	Bit
		-3	0
-2	1
		-1	2
0	3
		1	4
2	5
		3	6

For PAM4 encoded signals, the Tx taps may be identified as follows:

Pre2＝-2

Pre1＝-1

Main＝0

Post1＝1

Post2＝2.

however, other types of formatting may be used. A request to change an unsupported tap may be considered a protocol error and may be ignored.

If a link occurs before the protocol is used, both parties are considered to agree on the number and mapping of SerDes lanes used by the port.

In some examples, the message format for SETUP may include: { SETUP, Port _ id, < supported _ Taps >.

To identify the taps that are the subject of a request or response, the following identification data may be included in the complete request or response message:

{ Port _ id, Logical _ lane _ id, Tap _ id, REQ/RSP }, where:

port _ id is the local Port _ id received in the setup message

Logical _ lane _ id is a value from 0 to 15 indicating a Logical channel on a port

Tap _ id is the object Tap number (e.g., -3 to +3)

SETUP/REQ/RSP/TRAINED (0-3) identifies the message as a SETUP (0), request (1), response (2), or TRAINED (3) message. The trained message indicates completion of the protocol.

In some examples, the request may be one of: HOLD, INC, DEC. In some examples, the message format for REQUESTs may be: { REQ, Port _ id, Logical _ lane _ id, Tap _ id, < INC/DEC/HOLD > }. In some examples, the response may include one of: NOT _ UPDATED, MIN, MAX. In some examples, the message format for the request may include: { RSP, Port _ id, Logical _ lane _ id, Tap _ id, < NOT _ UPDATED/UPDATED/MIN/MAX > }. Accordingly, the complete message may contain the following information:

{SETUP,Port_id,<supported_Taps>,DESTRUCTIVE_MODE_ABILITY,DESTRUCTIVE_MODE_REQ,<random-initiator-bit>}

{REQ,Port_id,Logical_lane_id,Tap_id,<INC/DEC/HOLD>}

{RSP,Port_id,Logical_lane_id,Tap_id,<NOT_UPDATED/UPDATED/MIN/MAX>}

the transmitter may take time to complete a setting change (request) and the receiver may take time to know whether a configuration change has been applied or not (response). Because messages may be lost or corrupted, the exchange protocol may recover from the error.

Some SerDes may not be fully optimized incrementally and a re-adaptation mode may be used when a tap change (Rx equalization) is being evaluated, which drops the link in one or both directions. Support for this mode of operation may involve using a protocol that operates in one direction at a time. For example, one side may make a tap request and adjust its receiver equalization, and next, the other end of the link may begin the process for adjusting the receiver equalization. It is necessary to distinguish the cause of the link loss, which may be due to receiver adaptation (caused by the far-end party) or the requested tap change (caused locally).

The "destructive mode" (or "restart inhibit mode") may fail the link ("drop") during training, but the request receiver does not allow the link to drop until after a certain amount of time has elapsed, so that the default Tx tap settings are not restored until after the amount of time has elapsed (maximum link loss time) to avoid protocol restart. Training tap settings applied during successful retraining and before a time expiration are used and the training protocol may not be restarted. However, after the amount of time has elapsed, protocol training is resumed to determine that the Tx tap settings and/or default Tx tap settings may be used.

The operation in "destructive mode" is conveyed in the setup message by three parameters:

DESTRUCTIVE_MODE_ABILITY

DESTRUCTIVE_MODE_REQ

<max-link-loss-time>

DESTRUCTIVE _ MODE _ ABILITY indicates that the node supports DESTRUCTIVE MODE operation as an option.

DESTRUCTIVE _ MODE _ REQ indicates that the node needs (rather than just "requests") DESTRUCTIVE MODE operation.

< max-link-loss-time > is a 32-bit unsigned value that represents the maximum time (in milliseconds) that the link may be dropped and no protocol restart is performed. This time may be different in each direction and should be set to the maximum amount of time that local receiver adaptation should be allowed to take.

By setting default _ MODE _ REQ equal to 1 but default _ MODE _ identify equal to 0, a protocol error may be generated.

If one party needs the DESTRUCTIVE MODE (DESTRUCTIVE _ MODE _ REQ ═ 1) and the other party does not support the DESTRUCTIVE MODE (DESTRUCTIVE _ MODE _ abort ═ 0), the protocol may terminate.

If one or both parties require destructive mode operation, an additional message exchange is used to determine which party is in advance.

{INITIATOR_BID,<local-bid>,<remote-bid>}

< local-bid > is a non-zero 32-bit unsigned value that determines which party starts first. It may be generated by some random process to ensure convergence. The party with the largest "bid" precedes. In the event of a tie, each party submits a new random "bid".

< remote-bid > is the last received 32-bit value from the other party as its "bid". The initial value of this field is "0" before any bids are received from the other party. The INITIATOR _ BID message may be sent periodically to recover from lost messages.

Bidding continues until a bid is received from the other party, wherein the < remote-bid > field matches the value of the current < local-bid >, and wherein different values are used for local and remote bidding. At this point, both parties determine which one should look ahead, and the "winning" party can initiate a tap request.

In some examples, LLDP Protocol Data Units (PDUs) are used for capability advertisement, request, and response. In some examples, there are 3 types of TLVs:

StartUpTlv may include a local port identifier that the link partner will return in any subsequent TLV to identify the particular port. StartUpTlv may also include an indication of which taps are supported on the local device. The taps supported may be pre2, pre1, main, post1, post2, or fewer, more, or other taps. In some examples, InfiniBand link settings may be adjusted.

The RequestTlv may include the localPortId received in StartUpTlv for the port. RequestTlv may also include a logical LaneID (0-7) within localported, a tapId indicating which Tx tap is the subject of the request, and the request itself (HOLD 0 or UPDATE +/-1).

StatusTlv may include the localPortId received in StartUpTlv for the port. StatusTlv may be a logical LaneID (0-7) within localported, a tapId indicating which Tx tap is the object of the status report, and the status itself (NOT _ update ═ 0, update ═ 1, MIN ═ 2, MAX ═ 3). MIN or MAX may be returned to indicate that further changes in the same direction are not possible and should be considered the same as UPDATED.

In some examples, an LLDP PDU may have up to two link training TLVs per logical lane: one requestTlv and one statusstlv (responsetlv). The requestTlv may initiate a change in Tx tap settings on the link partner. The PDU may be generated in response to receiving the PDU from the link partner or after a timeout period. If any taps are in the process of being changed and a status tlv has not been received for it for the last certain amount of time, a PDU may be generated to duplicate the requested change to handle the missing or corrupted PDU.

In some examples, changes to the tap settings may be made with the smallest possible increment/decrement to minimize the tendency for link loss due to tap changes.

Fig. 2B depicts an example system for communicatively coupling a network device to another network device. For example, host 250 and device 232 may include network devices such as one or more of the following: a network interface, switch, router, server, host computing platform, interconnect, fabric, rack, or any computing or communication device. For example, the device 232 may be connected to an interface with multiple electrical links (e.g., backplane or copper cables). The system provides multiple channels of transmit-receive pairs that may be used to transmit or receive electrical signals between host 250 and device 232. The channel may send and/or receive signals. The transmitter of the channel may use an equalizer implemented in analog circuitry to generate an electrical signal for transmission. The equalizer may have one or more current sources to create a signal, whereby the weights of the current sources may be adjusted to change the signal characteristics. The equalizer settings may be modified to change the weights of the current sources. For example, a digital-to-analog converter (DAC) may be used to create a signal in the digital domain and output the result in an analog format.

Various embodiments use any microcontroller 244 to negotiate the time for completion of the link training and whether to extend the training time. In addition, microcontroller 244 can initiate and manage retraining of transmitter and/or receiver equalizer settings.

The transceiver 238 can be used for the transmission and reception of electrical signals between the device 232 and the host network interface device 250. The transceiver 238 may provide a plurality of transmit and receive channels for the communication of electrical signals between the device 232 and the host device 250. For example, channels 240-0 through 240-N may provide transmit and receive circuitry for coupling with receive and transmit circuitry of channels 254-0 through 254-N of host device 250. Lanes 240-0 through 240-N may provide serializer/deserializer (SerDes) formatting of signals. In some examples, the transceiver 238 may be part of a PMD or PHY.

Device 232 may be communicatively coupled to host device 250 through interconnect 242. The interconnect 242 may be an electrical signal conductor that couples the pins or holes of the channels 240-0 through 240-N of the pluggable device 232 to the holes or pins of the channels 254-0 through 254-N of the host 250. Host network interface device 250 can send or receive signals in electronic format to or from device 232.

Host device 250 may include a transceiver 252 for communicating with device 232. The transceiver 252 may include channels 254-0 through 254-N, where any of the channels 254-0 through 254-N include receive and transmit circuitry. In some examples, transceiver 252 may be part of a PMD or PHY. Any of the microcontrollers 256-0 to 256-N may be used to manage the operation of its channels according to the embodiments described herein.

In some embodiments, a single microcontroller may manage equalizer settings for one or more channels. The one or more parameters may cause a receiver or transmitter device in any of the paths 254-0 through 254-N to adjust its equalizer settings with respect to a particular tap, whether to increase or decrease coefficient values for the equalizer taps. In some embodiments, the setting of a tap may be adjusted independently of the adjustment of the setting of another tap.

In some examples, host 250 may request to change equalizer settings for any taps of the transmitter equalizer circuit of device 232. Similarly, device 232 may request to change the equalizer settings of any taps of the transmitter equalizer circuit of host 250. Accordingly, device 232 and host 250 may adjust transmitter equalizer settings used by the partner device. In addition, either of device 232 and host 250 may adjust receiver equalizer settings to compensate for channel distortion.

For example, to initiate an equalizer setting change, any of the microcontrollers 244-0 through 244-N may determine the signal quality of the received signal and determine which transmitter side tap of the host device 250 is to be changed and whether to increment or decrement the setting of the tap. For example, the eye opening of the received signal may be measured. The eye diagrams may represent 1 to 0 and 0 to 1 transitions of the signal and indicate whether or not a transition occurs in an isolated time region. The microcontroller may estimate inter-symbol interference (ISI) and select a setting based on the ISI that reaches a minimum value. The microcontroller may search for available transmitter tap settings and select the setting that results in the most open eye. The transmitter equalizer settings may be changed periodically at or after link startup and may be run periodically. Similar operations may be performed for microcontrollers 256-0 through 256-N to adjust the transmit equalizer settings of device 232.

Either device 232 or host 250 may perform packet processing such as one or more of the following: medium access control, any protocol layer processing, security, routing, destination lookup, etc.

Fig. 4A depicts a process sequence for link training. At 402, AN IEEE802.3 clause 73AN base page exchange may begin. In this example, 200Gbps link speed and RS-FEC capability are advertised. At 404, a next page swap may occur. In some examples, two link partners advertise that the provider OUI is identified as Intel, while one link advertises a training extension time of (up to) 5 seconds, and the other link advertises a training extension time of (up to) 10 seconds. However, the vendor OUI of the device may be advertised. In some examples, the highest common denominator (i.e., 5 seconds) of the extension time is selected. At 406, the link training scheme uses a default link training time of +5 seconds. An example link training format is in clause 72 of IEEE 802.3-2018.

Fig. 4B depicts an example sequence that may be used for link retraining operations. At 450, auto-negotiation may be applied to set one or more parameters for operation between partner devices. For example, IEEE802.3 clause 73AN may be used to set at least link speed, FEC mode, pause capability, and/or other parameters. At 452, link training may be performed to set transmit and receive SerDes equalizer parameters. For example, IEEE802.3 clause 72 link training may be performed to set transmit and/or receiver equalizer settings. In some cases, the link training duration may be extended according to examples described herein. Thereafter, data or other content may be transmitted across one or more channels or links.

At 454, the ability to retrain the link may be negotiated. For example, the transmitter may advertise in an initial setup message any taps for the link partner that can be adjusted by link retraining. The setup message identifies the local Port _ id that will be the subject of the tap change request by the link partner. At 456, a request to modify transmitter component settings can be received at the first device. The request may include a port identifier, a lane identifier, an object tap, or an increment/decrement/hold tap setting that will be the subject of a tap change request by the link partner. At 458, a response is provided to the second device to indicate that the specified tap settings have been applied. The second device may measure signal characteristics (e.g., BER, eye size, and other errors). In this example, the second device determines that the settings are acceptable. At 460, the second device indicates that the setting is to be maintained. Tap settings may be applied and stored for current and future use.

Note that both the first and second devices may make independent transmitter adjustments. In some examples, receiver equalizer settings may be set by the link partner in a manner similar to the manner in which transmitter tap settings are specified.

Fig. 5A depicts an example of an equalizer. The transmitter equalizer 500 includes a front body tap c (-1), a body tap c (0), and a rear body tap c (1). In the illustrative embodiment, the filter tap setting uses two bits to identify one of four possible tap values for the front body tap c (-1) and one of six possible tap values for the back body tap c (1). The volume tap c (0) coefficient may be calculated based on the values of the other two tap coefficients c (-1) and c (1) such that the equation is satisfied: c (0) -c (1) -c (-1) ═ 1. The filter tap settings may be modified by incrementing or decrementing the filter tap settings.

Fig. 5B depicts a functional model of the structure of a four-tap feed-forward equalizer (FFE)550 in a transmitter. FFE 550 is implemented in each communication channel interface of a chip-to-chip or chip-to-module interface. FFE 550 includes a front body tap c (-2), a front body tap c (-1), a body tap c (0), and a back body tap c (1). The filter tap settings may be modified by incrementing or decrementing the filter tap settings. The coefficients of any tap may be modified independently of the coefficients of the other one or more taps.

Fig. 6A depicts an example process for potentially extending the time allocated for link training. A transceiver in a first device having a wired or wireless connection to a transceiver in a second device performs a process. For example, the first device may be a network interface, a host device, an electrical or optical module, or any device. For example, the second device may be a network interface, a host device, an electrical or optical module, or any device. The connection may be a copper cable, an optical cable, a backplane, any type of ethernet cable, or any wired or wireless signal propagation medium. The transceiver may include a transmitter and a receiver. The signals propagated over the connection may be compatible with ethernet, fibre channel, InfiniBand, or serial attached small computer system interface (SAS).

At 602, an exchange occurs between a transceiver of a first device and a transceiver of a second device to determine common capabilities with a link partner. For example, AN IEEE802.3 AN may be implemented to determine at least link speed, FEC capabilities, and other capabilities. At 602, an operating mode for a connection may be selected. The operating mode may be determined to be the lowest capability advertised by the two transceivers. The operating modes include one or more of speed, Forward Error Correction (FEC), pause capability, and/or other capabilities.

At 604, the transceiver may perform an exchange phase to advertise other capabilities. For example, the capability may be advertised using the auto-negotiated next page exchange phase. Other capabilities may include the amount of extended link training time supported, unsupported, extended amount, etc.

At 606, the two transceivers may determine an amount to extend the link training time, if any, based on the capabilities of the transceivers and the advertised capabilities of the link partner transceivers to extend the link training time. For example, if one transceiver supports extended link training and the other does not, a default link training time is used. If one transceiver supports a higher extended link training time than the other transceiver, then the two transceivers use the highest common denominator of the extended link training times.

At 608, the transceiver conducts a link training phase with respect to the link training time plus an amount to extend the link training time. The link training phase may include: the requesting peer sends a change in equalization settings, monitors the impact on link quality at the receiver, and adjusts the equalizer settings to optimize one or more of errors, eye size, etc.

Fig. 6B depicts an example process for performing link retraining. At 650, the capability to perform link retraining is communicated to the link partner. For example, the capability to perform link retraining may be in the format of a setup message described herein. For example, 650 may include 652, wherein the capability to perform link retraining may include an indication to establish retraining, a port identifier for identification of taps that may be configured during link retraining.

At 654, a request may be received to reconfigure components available for configuration during link retraining. For example, the request may include a port identifier, a lane identifier, an object tap, or an increment/decrement/hold tap setting that will be the subject of a tap change request by the link partner.

At 656, it may be determined whether a response to the configuration request is received. For example, the response may include an indication of whether to perform the reconfiguration. Other examples of response messages are provided herein. For example, the response may include one or more of the following: an identifier of a port identifier, a channel identifier, an object of a tap, and updated/not updated/minimum reached/maximum reached. If a response is received, the process continues to 658. If no response is received, 656 may be repeated.

At 658, a determination may be made as to whether the configuration is acceptable. For example, a configuration may be determined to be acceptable if a Bit Error Rate (BER), an eye size, a signal-to-noise ratio (SNR), or any other desired characteristic is achieved. After determining that the configuration is acceptable, data transmission may continue or resume. Otherwise, the receiver may indicate other configurations at 654 and the process continues.

Fig. 7A depicts an example of states that may be used by a state machine that may be used to modify tap values in an attempt to achieve agreement on updated states by both ends of a link. For example, a SerDes driver or microcontroller associated with a SerDes may implement a state machine. The Logical _ lane _ id may support an independent state machine for updating each supported tap. The state machine may track requests for tap setting changes. In some examples, the status value in the response message may update the status. In some examples, the state machine may include three states:

ST _ IDLE, may initiate a changed state. Enter upon startup or after exiting the ST _ UPDATED state upon completion of a tap change.

ST _ UPDATE, enters when a requestTlv to the tap is issued. In this state, requestTlv changes to indicate UPDATE (+/-1). Exit upon receiving a status tlv indicating UPDATED (or LIMIT).

ST _ UPDATED, enters upon receipt of a status Tlv with a state UPDATED (or LIMIT). In this state, requestTlv is changed to indicate HOLD. Exit upon receiving status tlv indicating NOT _ UPDATED.

Request messages may be issued periodically to handle the case of message loss.

The initial state may be HOLD, where there is "no request for change". If a Request is made to change Tx tap settings and the state of the previous Request is NOT UPDATED, a Request may be issued and the state of the tap Request changed to Update INCREAMENT/DECREMENT depending on the Request. This state may be maintained until the state of the tap (from the response message) becomes one of UPDATED, MIN, or MAX. The UPDATED may indicate that a tap change request is fulfilled or performed. MIN may indicate that a decrement request was made and the tap is now at its minimum, or the tap is already at its minimum. Similarly, MAX may indicate that an increment request was made and the tap is now at its maximum, or the tap is already at its maximum.

FIG. 7B depicts an example state machine that can be used to track the state of an update request. For example, a SerDes driver or microcontroller associated with a SerDes may formulate a state machine. In some examples, each Logical _ lane _ id may support an independent state machine for updates to each supported tap. The initial state of the tap may be NOT UPDATED, so no change is in progress. When a request is received (INCREMENT(INC)/DECREMENT(DEC)), the state may be moved to the UPDATE TX TAP while a change is being effected in the hardware. After the change is completed, the state may be changed to one of three states depending on the result of the change. If the tap has been successfully changed, the state changes to UPDATED. If the tap is incremented and the tap is now at its maximum, the state may be changed to MAX. Similarly, if the tap is decremented and the tap is now at its minimum, the state is changed to MIN. This state may be maintained until a HOLD request is received, and after receiving the HOLD request, the state may be restored to NOT UPDATED.

Since it is possible that the change requested using this protocol could drop the link itself, each party to the link could restore its Tx taps to their original settings in the event of a link drop. Changing the transmitter side parameters may drop the link and in this case, the initial settings may be used before the adjustment.

Fig. 8 depicts a network interface that may be used with or by an embodiment. According to embodiments described herein, various resources in a network interface may perform link establishment, link training, or link retraining. In some examples, the network interface 800 includes a network interface, a network interface controller, or a network interface card. In some examples, the network interface 800 may be part of a switch or system on a chip (SoC) having a device (e.g., a processor or memory). Network interface 800 may include a transceiver 802, a processor 804, a transmit queue 806, a receive queue 808, a memory 810 and bus interface 812, and a DMA engine 852. The transceiver 802 may be capable of receiving and transmitting packets in compliance with an applicable protocol (e.g., ethernet as described in IEEE 802.3), although other protocols may be used. The transceiver 802 may receive packets from and transmit packets to a network via a network medium (not depicted). The transceiver 802 may include a PHY circuit 814 and a Media Access Control (MAC) circuit 816. PHY circuitry 814 may include encoding and decoding circuitry (not shown) to encode and decode data packets according to an applicable physical layer specification or standard. The MAC circuitry 816 may be configured to perform MAC address filtering on received packets, process MAC headers of the received packets by verifying data integrity, remove preambles and padding, and provide packet content for higher layer processing. The MAC circuitry 816 may be configured to assemble data to be transmitted into packets that include the destination and source addresses as well as network control information and error detection hash values.

Processor 804 may be any combination of processors, cores, Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), or other programmable hardware devices that allow for programming of network interface 800. For example, processor 804 may provide: resources for executing the workload are identified, and a bitstream for execution on the selected resources is generated. For example, an "intelligent network interface" may provide packet processing capabilities in a network interface using processor 804.

Packet allocator 824 may provide for distribution of received packets for processing by multiple CPUs or cores using slot allocation or RSS as described herein. When the packet distributor 824 uses RSS, the packet distributor 824 may compute a hash or make another determination based on the content of the received packet to determine which CPU or core is to process the packet.

Interrupt coalescing 822 may perform interrupt modulation whereby network interface interrupt coalescing 822 waits for multiple packets to arrive or for a timeout period to expire, and then generates an interrupt to the host system to process the received packet. Receive segment merging (RSC) may be performed by the network interface 800, whereby portions of an incoming packet are combined into segments of the packet. The network interface 800 provides the merged packet to the application.

A Direct Memory Access (DMA) engine 852 may copy packet headers, packet payloads, and/or descriptors directly from host memory to a network interface, or vice versa, rather than copying packets to an intermediate buffer at the host and then using another copy operation from the intermediate buffer to the destination buffer.

The memory 810 may be any type of volatile or non-volatile memory device and may store any queues or instructions to program the network interface 800. The send queue 806 may include data or references to data for transmission by the network interface. Receive queue 808 may include data or references to data received by a network interface from a network. Descriptor queue 820 may include descriptors that reference data or packets in transmit queue 806 or receive queue 808. Bus interface 812 may provide an interface with a host device (not depicted). For example, the bus interface 812 may be compatible with PCI, PCI Express, PCI-x, Serial ATA, and/or USB compatible interfaces (although other interconnect standards may be used).

In some examples, the network interfaces and other embodiments described herein may be used in connection with base stations (e.g., 3G, 4G, 5G, etc.), macro base stations (e.g., 5G networks), pico stations (e.g., IEEE 802.11 compliant access points), nano stations (e.g., for point-to-multipoint (PtMP) applications), indoor data centers, outdoor data centers, edge network elements, fog network elements, and/or hybrid data centers (e.g., data centers that use virtualization, cloud, and software-defined networking to deliver application workloads across physical data centers and distributed multi-cloud environments).

Fig. 9 depicts an example switch. Various embodiments may be used in or with a switch to perform link establishment, link training, or link retraining according to embodiments described herein. Switch 904 may route packets or frames in any format or according to any specification from any of ports 902-0 through 902-X to any of ports 906-0 through 906-Y (or vice versa). Any of the ports 902-0 to 902-X may be connected to a network of one or more interconnected devices. Similarly, any of ports 906-0 through 906-X may be connected to a network of one or more interconnected devices. Switch 904 may use a table that maps packet characteristics with associated output ports to determine to which port to transmit a packet or frame. For example, a matching action table may be used whereby a hash of a portion of a packet is used as an index to find an entry. Further, switch 904 may perform packet replication for forwarding packets or frames to multiple ports and queuing and then transmitting packets or frames to output ports. Some embodiments implement hash lookups in the P4 programming language (which is a programming language designed to allow for programming of packet forwarding in the data plane). In contrast to general-purpose languages (e.g., C or Python), P4 is a domain-specific language with many constructs optimized around network data forwarding.

Fig. 10 depicts a system. According to embodiments described herein, a system may use embodiments described herein to perform link establishment, link training, or link retraining. System 1000 includes a processor 1010 that provides processing, operation management, and execution of instructions for system 1000. Processor 1010 may include any type of microprocessor, Central Processing Unit (CPU), Graphics Processing Unit (GPU), processing core, or other processing hardware, or combination of processors, for providing processing for system 1000. Processor 1010 controls the overall operation of system 1000 and may be or include one or more programmable general-purpose or special-purpose microprocessors, Digital Signal Processors (DSPs), programmable controllers, Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), and the like, or a combination of such devices.

In one example, the system 1000 includes an interface 1012 coupled to the processor 1010, which may represent a higher speed interface or a high throughput interface for system components (e.g., the memory subsystem 1020 or the graphics interface component 1040 or the accelerator 1042) that require higher bandwidth connections. Interface 1012 represents an interface circuit, which may be a separate component or integrated onto the processor die. Where present, graphical interface 1040 interfaces graphical components for providing a visual display to a user of system 1000. In one example, graphical interface 1040 may drive a High Definition (HD) display that provides output to a user. High definition may refer to displays having a pixel density of approximately 100PPI (pixels per inch) or higher, and may include, for example, full high definition (e.g., 1080p), retinal displays, 4K (ultra high definition or UHD), or other formats. In one example, the display may comprise a touch screen display. In one example, the graphical interface 1040 generates a display based on data stored in the memory 1030 or based on operations performed by the processor 1010, or both. In one example, the graphical interface 1040 generates a display based on data stored in the memory 1030 or based on operations performed by the processor 1010, or both.

The accelerators 1042 may be fixed function offload engines that may be accessed or used by the processors 1010. The accelerator 1042 may be coupled to the processor 1010 using a memory interface (e.g., DDR4 and DDR5) or using any networking or connection standard described herein. For example, accelerators among accelerators 1042 may provide sequential and speculative decode operations, compression (DC) capabilities, cryptographic services (e.g., Public Key Encryption (PKE), cryptography, hashing/authentication capabilities, decryption, or other capabilities or services) in the manner described herein. In some embodiments, an accelerator among accelerators 1042 additionally or alternatively provides the field selection controller capabilities described herein. In some cases, accelerator 1042 may be integrated into a CPU socket (e.g., a connector to a motherboard or circuit board that includes the CPU and provides an electrical interface to the CPU). For example, accelerators 1042 may include single or multi-core processors, graphics processing units, logic execution units single or multi-level caches, functional units usable for independent execution of programs or threads, Application Specific Integrated Circuits (ASICs), Neural Network Processors (NNPs), programmable control logic, and programmable processing elements (e.g., Field Programmable Gate Arrays (FPGAs)). The accelerators 1042 may provide a plurality of neural networks, CPUs, processor cores, general purpose graphics processing units, or the graphics processing units may be made available to Artificial Intelligence (AI) or Machine Learning (ML) models. For example, the AI model may use or include any one or combination of: reinforcement learning scheme, Q learning scheme, deep Q learning or asynchronous dominant actor critics (A3C), combinatorial neural networks, recurrent combinatorial neural networks, or other AI or ML models. Multiple neural networks, processor cores, or graphics processing units may be made available for use with the AI or ML models.

Memory subsystem 1020 represents the main memory of system 1000 and provides storage for code to be executed by processor 1010 or data values to be used in executing routines. Memory subsystem 1020 may include one or more memory devices 1030 (e.g., Read Only Memory (ROM), flash memory, one or more variations of Random Access Memory (RAM) (e.g., DRAM), or other memory devices, or a combination of these devices). Memory 1030 stores and hosts, among other things, an Operating System (OS)1032 to provide a software platform for executing instructions in system 1000. Further, applications 1034 may be executed on the software platform of OS 1032 from memory 1030. The application 1034 represents a program having its own operating logic for performing the execution of one or more functions. Process 1036 represents an agent or routine that provides ancillary functionality to OS 1032 or one or more applications 1034 or combination. OS 1032, applications 1034 and processes 1036 provide software logic to provide functionality for system 1000. In one example, memory subsystem 1020 includes memory controller 1022, which is a memory controller used to generate and issue commands to memory 1030. It is to be appreciated that the memory controller 1022 may be a physical part of the processor 1010 or a physical part of the interface 1012. For example, memory controller 1022 may be an integrated memory controller integrated onto a circuit with processor 1010.

In some examples, processor 1010 may execute a device driver (not depicted) for network interface 1050. OS 1032 may determine the capabilities of network interface 1050 from the device drivers. For example, OS 1032 may receive an indication of the capabilities of network interface 1050 to perform one or more of the following capabilities or capabilities described herein: link training time is extended, link training is started earlier than scheduled, default link training time is changed or set, link retraining or component parameter modification. OS 1032 may request that a device driver enable or disable network interface 1050 to perform any of the capabilities described herein. In some examples, OS 1032 may itself enable or disable network interface 1050 to perform any of the capabilities described herein. OS 1032 may provide requests (e.g., from applications 1034) to network interface 1050 to utilize one or more capabilities of network interface 1050. For example, any application 1034 may request, through the network interface 1050, the use or non-use of any of the capabilities described herein. In some examples, a data center administrator may configure the network interface 1050 to perform any of the capabilities described herein.

Although not specifically shown, it is understood that system 1000 may include one or more buses or bus systems (e.g., memory bus, graphics bus, interface bus, or others) between the devices. A bus or other signal line may communicatively or electrically couple the components together or both. A bus may include physical communication lines, point-to-point connections, bridges, adapters, controllers, or other circuits or combinations. The bus may include, for example, one or more of the following: a system bus, a Peripheral Component Interconnect (PCI) bus, an ultra-transport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (Firewire).

In one example, system 1000 includes an interface 1014, which can be coupled to interface 1012. In one example, interface 1014 represents interface circuitry, which may include individual components and integrated circuits. In one example, a plurality of user interface components or peripheral components or both are coupled to the interface 1014. Network interface 1050 provides system 1000 with the ability to communicate with remote devices (e.g., servers or other computing devices) over one or more networks. Network interface 1050 may include an ethernet adapter, a wireless interconnect component, a cellular network interconnect component, USB (universal serial bus), or other wired or wireless standard-based or proprietary interface. The network interface 1050 may transmit data to devices or remote devices in the same data center or rack, which may include: the data stored in the memory is transmitted. The network interface 1050 may receive data from a remote device, which may include: the received data is stored in a memory. Various embodiments may be used in conjunction with the network interface 1050, the processor 1010, and the memory subsystem 1020.

In one example, system 1000 includes one or more input/output (I/O) interfaces 1060. The I/O interface 1060 may include one or more interface components (e.g., audio, alphanumeric, tactile/touch, or other interface) for user interaction with the system 1000. The peripheral interface 1070 may include any hardware interface not specifically mentioned above. A peripheral device generally refers to a device that is dependently connected to the system 1000. A dependency connection is a connection where the system 1000 provides a software platform or a hardware platform or both that perform operations and interact with a user.

In one example, system 1000 includes storage subsystem 1080 to store data in a nonvolatile manner. In one example, at least certain components of storage 1080 may overlap with components of memory subsystem 1020 in certain system implementations. Storage subsystem 1080 includes a storage device 1084, which may be or may include any conventional medium for storing large amounts of data in a non-volatile manner (e.g., one or more magnetic, solid-state, or optical-based disks or combinations). The storage 1084 stores the code or instructions and data 1086 in a persistent state (e.g., values are retained despite interruption of power to the system 1000). While memory 1030 is typically executing or operating on memory to provide instructions to processor 1010, storage 1084 may generally be considered "memory". While storage 1084 is non-volatile, memory 1030 may include volatile memory (e.g., if power is interrupted to system 1000, the value or state of the data is ambiguous). In one example, storage subsystem 1080 includes a controller 1082 for interfacing with a storage 1084. In one example, controller 1082 is either interface 1014 or a physical part of processor 1010, or may comprise circuitry or logic in both processor 1010 and interface 1014.

Volatile memory is memory whose state (and thus the data stored therein) is ambiguous if power is interrupted to the device. Dynamic volatile memory may involve: the data stored in the device is refreshed to maintain the state. One example of dynamic volatile memory includes DRAM (dynamic random access memory) or some variant (e.g., synchronous DRAM (sdram)). Examples of volatile memory include cache. The memory subsystem described herein may be compatible with a variety of memory technologies such as: DDR3 (double data rate version 3, original release by JEDEC (joint electronic device engineering council) on 27 th 6 th 2007), DDR4(DDR version 4, initial specifications published by JEDEC on 9 th 2012), DDR4E (DDR version 4), LPDDR3 (low power DDR version 3, JESD209-3B, JEDEC on 8 th 2013), LPDDR4 (dr lpdd version 4, JESD209-4, originally published by JEDEC on 8 th 2014), WIO2 (wide input/output version 2, JESD229-2, originally published by JEDEC on 8 th 2014), HBM (high bandwidth memory, JESD325, originally published by JEDEC on 10 th 2013, lpdr 5 (currently in discussion of JEDEC), HBM2(HBM version 2), HBM2(HBM version 2, currently in discussion of JEDEC), or other derived technologies based on these or extensions thereof.

Non-volatile memory (NVM) devices are memories that are explicit even to the device interrupting power states. In one embodiment, the NVM device can include a block addressable memory device (e.g., NAND technology), or more specifically, a multi-threshold level NAND flash memory (e.g., single level cell ("SLC"), multi-level cell ("MLC"), four-level cell ("QLC"), three-level cell ("TLC"), or some other NAND). The NVM devices may further include byte-addressable write-to-bit three-dimensional cross-point memory devices or other byte-addressable write-to-bit NVM devices (also referred to as persistent memory) (e.g., single or multi-level Phase Change Memory (PCM) or phase change memory with Switches (PCMs), NVM devices using chalcogenide phase change materials (e.g., chalcogenide glass), resistive memory including metal oxide based, oxygen vacancy based, and conductive bridge random access memory (CB-RAM), nanowire memory, ferroelectric random access memory (FeRAM, FRAM), Magnetoresistive Random Access Memory (MRAM) incorporating memristor technology, Spin Transfer Torque (STT) -MRAM, spin electron magnetic junction memory based devices, Magnetic Tunnel Junction (MTJ) based devices, DW (magnetic) and SOT (spin orbit domain wall transfer) based devices, spin transfer memory devices, Thyristor-based memory devices, or combinations of any of the above, or other memories)

A power source (not depicted) provides power to the components of the system 1000. More specifically, the power source typically interfaces with one or more power sources in the system 1000 to provide power to the components of the system 1000. In one example, the power supply includes an AC to DC (alternating current to direct current) adapter to plug into a wall outlet. The AC power may be a renewable energy (e.g., solar) power source. In one example, the power source includes a DC power source (e.g., an external AC-to-DC converter). In one example, the power source or power source includes wireless charging hardware to charge via the vicinity of a charging field. In one example, the electrical power source may include an internal battery, an alternating current power source, a motion-based power source, a solar power source, or a fuel cell source.

In an example, system 1000 may be implemented using interconnected computing bases (slids) of processors, memory, storage, network interfaces, and other components. High speed interconnects between components may be used, for example: ethernet (IEEE 802.3), Remote Direct Memory Access (RDMA), InfiniBand, Internet Wide area RDMA protocol (iWARP), fast UDP Internet connection (QUIC), RDMA over converged Ethernet (RoCE), peripheral component interconnect express (PCIe), Intel QuickPath interconnect (QPI), Intel HyperPath interconnect (UPI), Intel System on chip architecture (IOSF), Omnipath, compute cache Link (CXL), HyperTransport, Highcast, NVLink, Advanced Microcontroller Bus Architecture (AMBA) interconnect, OpenCAPI, Gen-Z, Accelerator Cache Coherent Interconnect (CCIX), 3GPP Long Term Evolution (LTE) (4G), 3GPP 5G, and variations thereof. Data may be copied or stored to the virtualized storage node using a protocol (e.g., NVMe over Fabrics (NVMe-af) or NVMe).

Embodiments herein may be implemented in various types of computing and networking equipment, such as switches, routers, racks, and blade servers (e.g., those employed in a data center and/or server farm environment). Servers used in data centers and server farms include array server configurations (e.g., rack-based servers or blade servers). These servers are interconnected by communication through various network provisions, such as dividing a collection of servers into Local Area Networks (LANs) with appropriate switching and routing facilities between the LANs to form a private intranet. For example, cloud hosting facilities may typically employ large data centers with a large number of servers. The blade includes a stand-alone computing platform (i.e., "server-on-card") configured to perform server-type functions. Accordingly, the blades include components common to conventional servers, including a main printed circuit board (motherboard) that provides internal wiring (e.g., a bus) for coupling appropriate Integrated Circuits (ICs) and other components mounted to the board.

Fig. 11 depicts an environment 1100 that includes a plurality of computing racks 1102, some of the computing racks 1102 including a rack top (ToR) switch 1104, a bay manager 1106, and a plurality of pooled system drawers. Various embodiments may be used in or with a switch to perform link establishment, link training, or link retraining according to embodiments described herein. Typically, the pooling system drawer may include a pooled computing drawer and a pooled storage drawer. Optionally, the pooling system drawer may further comprise a pooling storage drawer and a pooling input/output (I/O) drawer. In the illustrated embodiment, the pooling system drawer includesPooled computer drawers 1108 andATOM^TMpooled computer drawer 1110, pooled storage drawer 1112, pooled storage drawer 1114, and pool I/O drawer 1116. Some pool system drawers are connected to the ToR switch 1104 via a high-speed link 1118 (e.g., a 40 gigabit/second (Gb/s) or 100Gb/s ethernet link or a 100+ Gb/s silicon photonic (SiPh) optical link). In one embodiment, high-speed link 1118 includes an 800Gb/s SiPh optical link.

Multiple computer racks 1102 may be interconnected via their ToR switches 1104 (e.g., to a bay level switch or a data center switch), as shown by the connection to the network 1120. In some embodiments, the group of computer racks 1102 is managed as a pod via a pod manager 1106. In one embodiment, a single bay manager is used to manage the racks in the bay. Alternatively, a distributed cabin manager may be used for cabin management operations.

The environment 1100 also includes a management interface 1122 to manage various aspects of the environment. This includes managing the rack configuration, with corresponding parameters stored as rack configuration data 1124.

Various embodiments may be implemented using hardware elements, software elements, or a combination of both. In some examples, hardware elements may include devices, components, processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, ASICs, PLDs, DSPs, FPGAs, memory units, logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. In some examples, a software element may include a software component, a program, an application, a computer program, an application program, a system program, a machine program, operating system software, middleware, firmware, a software module, a routine, a subroutine, a function, a method, a procedure, a software interface, an API, an instruction set, computing code, computer code, a code segment, a computer code segment, a word, a value, a symbol, or any combination thereof. Determining whether to implement an example using hardware elements and/or software elements may vary, as desired, for a given implementation in accordance with any number of factors, such as desired computational rate, power levels, thermal tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints. Note that hardware, firmware, and/or software elements may be referred to herein, collectively or individually, as "modules" or "logic. A processor may be a hardware state machine, digital control logic, a central processing unit, or any combination of one or more hardware, firmware, and/or software elements.

Some examples may be implemented using or as an article of manufacture or at least one computer readable medium. The computer readable medium may include a non-transitory storage medium for storing logic. In some examples, a non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and so forth. In some examples, logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, APIs, instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

According to some examples, a computer-readable medium may include a non-transitory storage medium for storing or maintaining instructions that, when executed by a machine, computing device, or system, cause the machine, computing device, or system to perform a method and/or operations in accordance with the described examples. The instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like. The instructions may be implemented according to a predetermined computer language, manner or syntax, for instructing a machine, computing device or system to perform a certain function. The instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and/or interpreted programming language.

One or more aspects of at least one example may be implemented by representative instructions stored on at least one machine-readable medium representing various logic within a processor which, when read by a machine, computing device or system, causes the machine, computing device or system to fabricate logic to perform the techniques described herein. These representations, known as "IP cores" may be stored on a tangible, machine-readable medium and provided to various customers or manufacturing facilities for loading into the manufacturing machines that actually make the logic or processor.

The appearances of the phrase "one example" or "an example" are not necessarily all referring to the same example or embodiment. Any aspect described herein may be combined with any other or similar aspect described herein, whether or not such aspects are described with respect to the same figures or elements. Dividing, omitting, or including block functions depicted in the figures does not infer that the hardware components, circuits, software and/or elements for implementing these functions would be necessarily be divided, omitted, or included in embodiments.

Some examples may be described using the expression "coupled" and "connected" along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, descriptions using the terms "connected" and/or "coupled" may indicate that two or more elements are in direct physical or electrical contact with each other. The term "coupled," however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other.

The terms "first," "second," and the like, herein do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The terms "a" and "an" herein do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item. The term "assert" as used herein with reference to a signal means the state of the signal in which the signal is active and can be realized by applying any logic level, either a logic 0 or a logic 1, to the signal. The term "following" or "following" may refer to following immediately or following some other event or events. Other sequences of operations may also be performed according to alternative embodiments. Further, additional operations may be added or removed depending on the particular application. Any combination of variations may be used, and many variations, modifications, and alternative embodiments thereof will be apparent to those of ordinary skill in the art having the benefit of this disclosure.

A disjunctive language (e.g., the phrase "X, Y or at least one of Z") is understood within the context of being commonly used to refer to items, etc., as either X, or Y, or Z, or any combination thereof (e.g., X, Y and/or Z), unless specifically stated otherwise. Thus, this disjunctive language is not generally intended to, and should not, imply that particular embodiments require that at least one of X, at least one of Y, or at least one of Z occur for each. Additionally, conjunctive language (e.g., at least one of the phrases "X, Y and Z") should also be understood to mean X, Y, Z or any combination thereof, including "X, Y and/or Z," unless specifically stated otherwise.

Illustrative examples of the devices, systems, and methods disclosed herein are provided below. Embodiments of the apparatus, systems, and methods may include any one or more of the examples described below, and any combination thereof.

The flow diagrams illustrated herein provide examples of sequences of various process actions. The flow diagrams may indicate operations to be performed by software or firmware routines and physical operations. In one embodiment, the flow diagram may illustrate the state of a Finite State Machine (FSM) that may be implemented in hardware and/or software. Although shown in a particular sequence or order, the order of the acts may be modified unless otherwise specified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes may be performed in a different order, and some actions may be performed in parallel. Further, one or more acts may be omitted in various embodiments; thus, not all acts may be required in every embodiment. Other process flows are possible.

The various components described herein may be means for performing the operations or functions described. Each component described herein includes software, hardware, or a combination thereof. These components may be implemented as software modules, hardware modules, dedicated hardware (e.g., application specific hardware, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), etc.), embedded controllers, hardwired circuitry, and so forth.

Example 1 includes an apparatus comprising: a first device associated with a lane of a communication link, wherein: the first device comprises a transmitter and a receiver; the first device is to receive a first communication identifying a capability to retrain a link; the first device is to send a second communication identifying one or more components, and the second communication is to cause one or more parameters of the one or more components to be modified; and the first device is to receive a third communication identifying a retrained state.

Example 2 includes any example, wherein the one or more components comprise an equalizer and the one or more parameters comprise at least one tap setting.

Example 3 includes any example, wherein the one or more parameters comprise a precursor, subject, or postcursor equalizer setting.

Example 4 includes any example, wherein the first device is to: advertising one or more capabilities; receiving one or more capabilities from the second device; and selecting an operating mode for the channel based on the advertised and received capabilities.

Example 5 includes any example, wherein the one or more capabilities comprise one or more of a link speed, a Forward Error Correction (FEC) capability, or a pause capability.

Example 6 includes any example, wherein the first communication comprises a setup message identifying at least one tap adjustable through link retraining and comprising a local port identifier of a local port to be the subject of a tap change request by the first device.

Example 7 includes any example, wherein the second communication comprises a request message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, changing a tap setting, or maintaining a tap setting.

Example 8 includes any example, wherein the third communication comprises a response message including one or more of: a port identifier, a lane identifier, an identifier of one or more taps, an updated indication, an un-updated indication, an indication of reaching a minimum value, or an indication of reaching a maximum value.

Example 9 includes any example, wherein the first device is to: link retraining is initiated in response to a change in power, voltage, or temperature, or elapsed time.

Example 10 includes any example, and includes one or more of: a server, a rack, or a data center, and wherein the first device is disposed in one or more of: a server, a rack, or a data center.

Example 11 includes any example, and includes a method for link retraining, the method comprising: receiving, by a receiver in a first device, a signal from a transmitter in a second device over a channel, the signal including a first communication identifying a capability to retrain a link; sending a second communication from the first device comprising one or more components of a second device whose capabilities are to be adjusted and a request to modify one or more parameters of the one or more components; and receiving, at the first device, a third communication identifying a retrained state.

Example 12 includes any example, wherein the one or more components comprise an equalizer and the one or more parameters comprise at least one tap setting.

Example 13 includes any example, wherein the one or more parameters comprise a precursor, subject, or postcursor equalizer setting.

Example 14 includes any example, wherein the first communication comprises a setup message identifying at least one tap adjustable through link retraining and comprising a local port identifier of a local port to be the subject of a tap change request by the first device.

Example 15 includes any example, wherein the second communication comprises a request message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, changing a tap setting, or maintaining a tap setting.

Example 16 includes any example, wherein the third communication comprises a response message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, an updated indication, an un-updated indication, an indication of reaching a minimum value, or an indication of reaching a maximum value.

Example 17 includes any example, and includes: one lane at a time is trained whereby the first device makes a tap request and adjusts its receiver, and then the first device's link partner begins link retraining for its receiver.

Example 18 includes any example, and includes: link retraining is initiated in response to a change in power, voltage, or temperature, or elapsed time.

Example 19 includes any example, and includes a computer-readable medium having instructions stored thereon, which if executed by one or more processors, cause the one or more processors to: link retraining is performed by: receiving, by a receiver in a first device, a signal from a transmitter in a second device over a channel, the signal including a first communication identifying a capability to retrain a link; sending, from the first device, a second communication comprising one or more components whose capabilities are to be adjusted and a request to modify one or more parameters of the one or more components; and receiving, at the first device, a third communication identifying a retrained state.

Example 20 includes any example, wherein the one or more components comprise an equalizer and the one or more parameters comprise at least one tap setting.

Example 21 includes any example, wherein the first communication comprises a setup message identifying at least one tap adjustable through link retraining and comprising a local port identifier of a local port to be the subject of a tap change request by the first device.

Example 22 includes any example, wherein the second communication comprises a request message comprising one or more of: a port identifier, a lane identifier, an identifier of one or more taps, changing a tap setting, or maintaining a tap setting.