阅读说明：本技术 具有故障转移机制的otn多机箱系统中的自动时钟相位同步 (Automatic clock phase synchronization in OTN multi-chassis systems with failover mechanism ) 是由 尼沙·阿伊卡拉 皮尤什·杰州 罗希思·斯雷尼瓦桑 锡达林格普拉萨德·古里 巴萨瓦林加·库拉谢 于 2019-01-15 设计创作，主要内容包括：公开了一种用于由网络元件使用的方法,所述网络元件包括用于向一个或多个相邻网络元件传送公共时钟信号的多个端口。所述方法包括：对于多个端口中的至少一个端口：当所述端口处于预定义监听模式时,确定所述端口是否与相邻网络元件连接；以及当所述端口被连接时,计算所述网络元件和所述相邻网络元件之间的相应时钟相位延迟值。所述方法还包括基于从多个预定义角色分配给所述网络元件的角色,向所述端口分配时钟信号同步角色。所述网络元件被配置为使用所述相应时钟相位延迟值和使用所述时钟信号同步角色来传送所述公共时钟信号。(A method is disclosed for use by a network element comprising a plurality of ports for transmitting a common clock signal to one or more adjacent network elements. The method comprises the following steps: for at least one port of the plurality of ports: determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode; and calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected. The method also includes assigning a clock signal synchronization role to the port based on the role assigned to the network element from the plurality of predefined roles. The network element is configured to transmit the common clock signal using the respective clock phase delay values and using the clock signal synchronization role.)

1. A method for use by a network element comprising a plurality of ports for transmitting a common clock signal to one or more neighboring network elements, the method comprising:

for at least one port of the plurality of ports:

determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode;

calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected; and is

Assigning a clock signal synchronization role to the port based on a role assigned to the network element from a plurality of predefined roles,

wherein the network element is configured to transmit the common clock signal using the respective clock phase delay values and using the clock signal synchronization role.

2. The method of claim 1, further comprising:

for the at least one of the plurality of ports, after calculating the respective clock phase delay value, switching the port to the predefined listening mode.

3. The method of any preceding claim, wherein the network element is assigned a master role, and wherein the port is assigned a forward clock signal synchronization role for forwarding the common clock signal.

4. The method of any preceding claim, wherein calculating the respective clock phase delay value comprises:

generating a first reference clock signal having a frequency greater than the common clock signal;

starting a counter of the network element in response to starting to transmit the common clock signal to the neighboring network element, wherein the counter receives the first reference clock signal;

collecting a value of the counter in response to receiving the common clock signal looped back from the neighboring network element; and

calculating the respective clock phase delay value using the average acquisition value of the counters.

5. The method of claim 4, further comprising:

generating a second reference clock signal using a digital phase locked loop of the network element,

wherein the second reference clock signal is generated based on the first reference clock signal.

6. The method of any of claims 4 to 5, wherein collecting the value of the counter occurs as a first instance, the method further comprising:

performing collecting at least a second instance of the value of the counter in response to receiving the common clock signal looped back from the neighboring network element,

wherein the respective clock phase delay values are calculated using the values of the counters collected during the first and second instances.

7. The method of any of claims 1 to 3, further comprising:

receiving the common clock signal from the neighboring network element;

generating a first reference clock signal having a frequency greater than the common clock signal, wherein the first reference clock signal is frequency synchronized with the common clock signal;

in response to a positive edge of the common clock signal, beginning to decrement a first counter from a predefined value using the first reference clock signal;

in response to a negative edge of the common clock signal, beginning to decrement a second counter from the predefined value using the first reference clock signal;

generating a positive edge of a compensated clock signal when the first counter is decremented to a zero value; and

generating a negative edge of the compensated clock signal when the second counter is decremented to a zero value.

8. The method of claim 7, further comprising:

generating a second reference clock signal using a first digital phase locked loop of the network element,

wherein the second reference clock signal is generated based on the first reference clock signal.

9. The method of claim 8, wherein generating a positive edge of the compensated clock signal and generating a negative edge of the compensated clock signal are performed using a second digital phase-locked loop of the network element.

10. The method of any of claims 7 to 9, wherein the predefined value comprises: a first period corresponding to a frequency of the common clock signal divided by a second period corresponding to a frequency of the first reference clock signal.

11. A network element, comprising:

a plurality of ports for transmitting a common clock signal to one or more adjacent network elements; and

one or more computer processors coupled with the plurality of ports and configured to:

for at least one port of the plurality of ports:

determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode;

calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected; and is

Assigning a clock signal synchronization role to the port based on a role assigned to the network element from a plurality of predefined roles,

wherein the network element is configured to transmit the common clock signal using the respective clock phase delay values and using the clock signal synchronization role.

12. The network element of claim 11, wherein the one or more computer processors are further configured to:

for the at least one of the plurality of ports, after calculating the respective clock phase delay value, switching the port to the predefined listening mode.

13. The network element of any of claims 11 to 12, wherein the network element is assigned a master role, and wherein the port is assigned a forward clock signal synchronization role for forwarding the common clock signal.

14. The network element of any of claims 11 to 13, wherein calculating the respective clock phase delay value comprises:

generating a first reference clock signal having a frequency greater than the common clock signal;

starting a counter of the network element in response to starting to transmit the common clock signal to the neighboring network element, wherein the counter receives the first reference clock signal;

collecting a value of the counter in response to receiving the common clock signal looped back from the neighboring network element; and

calculating the respective clock phase delay value using the average acquisition value of the counters.

15. The network element of claim 14, wherein the one or more computer processors are further configured to:

generating a second reference clock signal using a digital phase locked loop of the network element,

wherein the second reference clock signal is generated based on the first reference clock signal.

16. The network element of any of claims 14 to 15, wherein collecting the value of the counter occurs as a first instance, and wherein the one or more computer processors are further configured to:

performing collecting at least a second instance of the value of the counter in response to receiving the common clock signal looped back from the neighboring network element,

wherein the respective clock phase delay values are calculated using the values of the counters collected during the first and second instances.

17. The network element of any of claims 11-13, wherein the one or more computer processors are further configured to:

receiving the common clock signal from the neighboring network element;

generating a first reference clock signal having a frequency greater than the common clock signal, wherein the first reference clock signal is frequency synchronized with the common clock signal;

in response to a positive edge of the common clock signal, beginning to decrement a first counter from a predefined value using the first reference clock signal;

in response to a negative edge of the common clock signal, beginning to decrement a second counter from the predefined value using the first reference clock signal;

generating a positive edge of a compensated clock signal when the first counter is decremented to a zero value; and

generating a negative edge of the compensated clock signal when the second counter is decremented to a zero value.

18. The network element of claim 17, wherein the one or more computer processors are configured to:

generating a second reference clock signal using a first digital phase locked loop of the network element,

wherein the second reference clock signal is generated based on the first reference clock signal.

19. The network element of claim 18, wherein generating a positive edge of the compensated clock signal and generating a negative edge of the compensated clock signal are performed using a second digital phase-locked loop of the network element.

20. The network element of any of claims 17 to 19, wherein the predefined value comprises: a first period corresponding to a frequency of the common clock signal divided by a second period corresponding to a frequency of the first reference clock signal.

21. A computer readable medium having computer readable program code embodied therein, the computer readable program code executable by one or more computer processors of a network element to perform operations comprising:

for at least one port of a plurality of ports for transmitting a common clock signal to one or more adjacent network elements:

determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode;

calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected; and is

Assigning a clock signal synchronization role to the port based on a role assigned to the network element from a plurality of predefined roles,

wherein the network element is configured to transmit the common clock signal using the clock phase delay value and using the clock signal synchronization role.

22. The computer readable medium of claim 21, wherein the computer readable program code is executable by the one or more computer processors to perform the method of any of claims 2 to 10.

Technical Field

Embodiments presented in this disclosure relate generally to high speed communication systems and, more particularly, to techniques for automatic clock phase synchronization between network elements in a communication system.

Background

In an aggregate transport architecture, ethernet frames (specifically layer 2 or L2 ethernet frames) are transported transparently using various Optical-based transport networks, such as Synchronous Optical Networks (SONET), Optical transport networks (OTNs, or Optical Channel Data Unit (ODU) -based packets), Dense Wavelength Division Multiplexing (DWDM), and so on.

The protocol of the aggregate transport architecture may require a phase-locked reference clock that is common to multiple network elements. For example, in the Packet structure based otn (otn Over Packet fabric) protocol, an 311.04 megahertz (MHz) common reference clock is phase locked to a common time stamp synchronization pulse of 8 kilohertz (kHz). Maintaining phase synchronization may be accomplished by controlling clock signal routing delay skew for individual network elements. However, for communication systems with multiple network elements, maintaining phase synchronization is more complex, which may require nanosecond accuracy in some cases. For example, factors such as component variations and/or temperature variations of different network elements and properties of wiring connecting different network elements may affect the amount of delay of the phase synchronization pulse.

Drawings

So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments.

Fig. 1 illustrates an exemplary network element according to embodiments described herein.

Fig. 2 is a diagram of an exemplary communication system according to embodiments described herein.

Fig. 3 is a diagram illustrating determining a delay calculation between two network elements according to embodiments described herein.

Fig. 4 is a diagram illustrating delay compensation between two network elements according to embodiments described herein.

Fig. 5 is an exemplary method for configuring a communication system including multiple network elements according to embodiments described herein.

Fig. 6 is an exemplary method for configuring a network element within a communication system according to embodiments described herein.

Fig. 7 is a diagram illustrating synchronization states of a communication system according to embodiments described herein.

Fig. 8-10 each illustrate an updated synchronization status of a communication system according to embodiments described herein.

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.

Detailed Description

SUMMARY

One embodiment presented by the present disclosure is a method for use by a network element that includes a plurality of ports for transmitting a common clock signal to one or more neighboring network elements. The method comprises, for at least one port of the plurality of ports: determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode; and calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected. The method also includes assigning a clock signal synchronization role to the port based on the role assigned to the network element from the plurality of predefined roles. The network element is configured to transmit the common clock signal using the clock phase delay value and using the clock signal synchronization role.

Another embodiment presented by the present disclosure is a network element comprising: a plurality of ports for transmitting a common clock signal to one or more adjacent network elements; and one or more computer processors coupled with the plurality of ports. The one or more computer processors are configured to, for at least one port of the plurality of ports: determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode; and calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected. The one or more computer processors are further configured to assign a clock signal synchronization role to the port based on the role assigned to the network element from the plurality of predefined roles. The network element is configured to transmit the common clock signal using the clock phase delay value and using the clock signal synchronization role.

Another embodiment presented by the present disclosure is a non-transitory computer-readable medium having computer-readable program code embodied therein, the computer-readable program code executable by one or more computer processors of a network element to perform operations. The operations include, for at least one port of a plurality of ports for communicating a common clock signal to one or more neighboring network elements: determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode; and calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected. The operations also include assigning a clock signal synchronization role to the port based on a role assigned to the network element from a plurality of predefined roles. The network element is configured to transmit the common clock signal using the clock phase delay value and using the clock signal synchronization role.

Example embodiments

In various embodiments disclosed herein, a network element includes a plurality of ports for communicating a common clock signal to one or more neighboring network elements, and one or more computer processors coupled with the plurality of ports. The one or more computer processors are configured to, for at least one port of the plurality of ports: determining whether the port is connected with a neighboring network element when the port is in a predefined listening mode; and calculating respective clock phase delay values between the network element and the adjacent network element when the port is connected. In some cases, this may be performed for each of a plurality of ports. The one or more computer processors are further configured to: assigning a clock signal synchronization role to the port based on the role assigned to the network element from the plurality of predefined roles. In some cases, this may also be performed for each of the plurality of ports.

In this manner, the network element may establish and/or maintain clock phase synchronization between multiple network elements with a desired accuracy (e.g., in microseconds or nanoseconds). In addition, a communication system including multiple network elements provides redundancy in the event of one or more link failures between the network elements.

Advantageously, no special packet protocols or complex software stacks are required to implement the clock phase synchronization techniques disclosed herein. Although relatively simple, clock phase synchronization techniques can achieve reasonably high accuracy while requiring minimal additional logic to be included in the network element. Redundancy between different network elements can be achieved without user configuration and fault handling can be performed automatically without user intervention.

Fig. 1 illustrates an exemplary network element 100 according to embodiments described herein. Although network element 100 is described herein as being configured to communicate using the packet-based fabric OTN protocol, operation of network element 100 using other protocols is also possible. Network element 100 may have any suitable implementation, such as a multi-layer aggregate transport network element configured to provide both packet-switched and optical-switched functionality.

Network element 100 generally includes one or more computer processors configured to route and/or process network traffic, and memory (which may include volatile and/or non-volatile memory elements). Network element 100 may also include various input/output (I/O) interfaces coupled with one or more computer processors, such as electrical and/or optical network interfaces, interfaces for displays or user input devices, and so forth.

The functionality of network element 100 may be implemented using hardware, software, firmware, or a combination thereof. For example, network element 100 may be implemented as a computing device mounted on a particular rack and/or chassis.

In the context of a packet-based fabric OTN protocol, network element 100 includes ingress OTN function 110, ingress segmentation and reassembly (SAR) function 115, packet fabric 120, egress SAR function 125, and egress OTN function 130. The network element 100 is configured to receive as input a first optical signal (OTUk stream 105) and to receive as output a second optical signal (OTUk stream 135).

The ingress SAR function 115 and the egress SAR function 125 are provided with a common clock signal (140; REFCLK) and a synchronization pulse signal (145; SYNC). The common clock signal 140 and the synchronization pulses 145 may be derived from a common timing reference (timing reference) and phase locked to each other. As described above, the packet structure based OTN protocol uses an 311.04MHz common clock signal 140 and an 8kHz common sync pulse signal 145, but other signal frequencies are possible.

The ingress OTN function 110 includes an OTUk framer (frame) configured to terminate OTUk overhead, perform Forward Error Correction (FEC), and output a corresponding higher order ODUk stream. In some embodiments, the ingress OTN function 110 further comprises one or more ODTUjk demultiplexer stages that extract the low order ODUj/ODUflex clients from the high order ODUk carrier streams.

Ingress SAR function 115 is configured to segment the high and/or low order streams into packets or cells that are forwarded to packet structure 120. The packet structure 120 switches the traffic which is then reassembled back into the ODU data stream processed by the egress OTN function 130 by the egress SAR function 125.

The egress OTN function 130 is configured to accommodate higher order ODUk flows and add OTUk overhead and FEC checksum bytes to form an OTUk flow 135. In some embodiments, the egress OTN function 130 is configured to multiplex the lower order ODUj/ODUflex streams into the higher order ODUk streams 135. In this manner, the network element 100 is configured to maintain the timing integrity of the ODU flow from ingress to egress.

Additional information regarding the functionality of network elements 100 using the OTN Protocol based on a packet structure is described in "OTN over packet Fabric Protocol (OFP) Implementation agent" published by the Optical internet Forum (Optical internet Forum) IA # OIF-OFP-01.0 in 11 months 2011, which is incorporated herein by reference. Further, alternative implementations of network element 100 may include suitable functionality for operating using one or more alternative protocols.

Fig. 2 is a diagram illustrating an exemplary communication system 200 according to embodiments described herein. Communication system 200 may be used in conjunction with other embodiments discussed herein. More specifically, communication system 200 includes a plurality of network elements 100-0, 100-1, 100-2, 100-3 (collectively network elements 100), each of which may include the functionality of network element 100 described with reference to fig. 1.

Each network element 100 includes four (4) Timing Extension (TE) ports 205, where each TE port 205 is configured to communicate a common clock signal to one or more adjacent network elements 100. As shown, the four TE ports 205 of each network element 100 are arranged into two pairs of TE ports 205-a first pair including the TE ports TE0-E, TE0-W and a second pair including the TE ports TE1-E, TE 1-W. Each TE port 205 in each pair includes a direction (or orientation) identifier that indicates the direction for transmitting the common clock signal, thereby indicating how the particular TE port 205 should be connected with the TE port 205 of the adjacent network element 100. In some embodiments, the direction identifier is selected from two predefined directions. As shown, the first pair includes TE port TE0-E having an "east" (E) direction identifier and TE port TE0-W having a "west" (W) direction identifier.

In some embodiments, multiple network elements 100 are communicatively coupled to form one or more clock loops 210-1, 210-2, wherein a common clock signal may be communicated to each of the network elements 100 using each of the clock loops 210-1, 210-2. In clock loop 210-1, TE1-E of network element 100-0 is connected to TE1-W of network element 100-1 using link 215-0, TE1-E of network element 100-1 is connected to TE1-W of network element 100-2 using link 215-1, TE1-E of network element 100-2 is connected to TE1-W of network element 100-3 using link 215-2, and TE1-E of network element 100-3 is connected to TE1-W of network element 100-0 using link 215-3.

In clock loop 210-2, TE0-E of network element 100-0 is connected to TE0-W of network element 100-1 using link 220-0, TE0-E of network element 100-1 is connected to TE0-W of network element 100-2 using link 220-1, TE0-E of network element 100-2 is connected to TE0-W of network element 100-3 using link 220-2, and TE0-E of network element 100-3 is connected to TE0-W of network element 100-0 using link 220-3.

Links 215-0, 215-1, 215-2, and 215-3 (collectively referred to as links 215) and links 220-0, 220-1, 220-2, and 220-3 (collectively referred to as links 220) may each include one or more electrical conductors arranged to carry a common clock signal between adjacent network elements 100. In one example, each TE port 205 includes an RJ45 connector (also referred to as an 8-position 8-contact (8P8C) connector) and each link 215, 220 includes category 6 (twisted pair) wiring. Other types of connectors and/or wiring are possible.

In each clock loop 210-1, 210-2, the common clock signal may be transmitted in either direction on a particular link 215, 220, depending on the configuration of the connected TE port 205. In some embodiments, each of the TE ports 205 may be assigned a clock signal synchronization role selected from a plurality of roles. Some non-limiting examples of clock signal synchronization roles include a "master" role, a "backup" master role, an "alternate" master role, a "forward" role, and a "disable" role.

In some embodiments, the TE ports 205 of the network element 100 may be dynamically configured and/or reconfigured in response to the link conditions of the various links 215, 220. The configuration and/or reconfiguration may be performed automatically by network element(s) 100 without user intervention. For example, TE ports 205 connected by links 215, 220 may have their clock signal synchronization roles adjusted in response to determining that links 215, 220 are degraded or failing. In this manner, the common clock signal may be dynamically rerouted to avoid one or more links 215, 220 that encounter performance problems.

Although four network elements 100 are illustrated, alternative implementations of communication system 200 may include alternative numbers of network elements 100. Further, although each network element 100 is shown with four TE ports 205, alternative implementations of the network element 100 may include alternative numbers of TE ports 205. Still further, while each of the TE ports 205 includes a direction identifier selected from two predefined directions (i.e., east or west), alternative implementations of the network element 100 may assign the direction identifiers to the TE ports 205 from more than two predefined directions. Still further, while two clock loops 210-1, 210-2 are illustrated, and each clock loop 210-1, 210-2 includes each of the network elements 100, alternative implementations of the communication system 200 may include alternative numbers of clock loops and/or different compositions of the network elements 100 within different clock loops.

Fig. 3 is a diagram 300 illustrating determining a delay calculation between two network elements according to embodiments described herein. The process illustrated by diagram 300 may be used in conjunction with other embodiments disclosed herein, for example, between adjacent network elements 100 of the communication system 200 illustrated in fig. 2.

Network element 100-0 includes logic 305-0 and network element 100-1 includes logic 305-1. The logic 305-0, 305-1 may be implemented using hardware, software, firmware, or a combination thereof. The logic 305-0, 305-1 may be implemented in any suitable form, for example in a control card installed in the respective network element 100-0, 100-1.

Logic 305-0 includes at least a first Digital Phase Locked Loop (DPLL)310-0 and a Field Programmable Gate Array (FPGA)315-0, and logic 305-1 includes at least a first DPLL 310-1 and an FPGA 315-1. Although not shown identically, logic 305-0 may include the same or similar functionality as logic 305-1. FPGA315-1 includes a delay calculation module 320-1 and a delay compensation module 325-1.

During a predetermined "listening" mode of the network element 100, for each of a plurality of TE ports, the network element 100 may determine whether the port is connected with a neighboring network element 100. During the listening mode, the neighboring network element 100 and its connected port may be identified using a clock presence interrupt received from the neighboring network element 100. When it is determined that a port is to be connected, network element 100 uses delay calculation module 320-1 to calculate a clock phase delay value for the link. The clock phase delay value is a function of at least the length of the wiring connecting network elements 100.

In some embodiments, to perform the clock phase delay value calculation, a software application executing on one of network elements 100-0, 100-1 assigns a master role to network element 100-1 and a slave role to network element 100-0. In some cases, assigning roles to different network elements 100-1, 100-2 may include sending control signals to one or both of network elements 100-1, 100-2.

In some embodiments, a common clock signal 335 (e.g., an 8kHz common sync pulse signal 145) is sent from the TE port TE0-W of the network element 100-1 to the TE port TE0-E of the network element 100-0.

DPLL 310-1 of network element 100-1 is configured to generate first reference clock signal 330 and to send first reference clock signal 330 to FPGA 315-1. First reference clock signal 330 may be highly stable and accurate according to the reference oscillator of DPLL 310-1. Based on the first reference clock signal 330, the FPGA315-1 internally generates a second reference clock signal having a higher clock frequency than the first reference clock signal 330.

In one non-limiting example, and corresponding to an 8kHz synchronization signal, the first reference clock signal 330 has a frequency of 25MHz and the second reference clock signal has a frequency of 200 MHz. Other frequencies of the first reference clock signal 330 and/or the second reference clock signal are also possible. Notably, the higher frequency of the second reference clock signal corresponds to increased accuracy of the clock phase delay calculation.

In some embodiments, FPGA315-1 includes a counter that receives a second reference clock signal. For example, the counter may comprise a 32-bit timestamp counter corresponding to a 200MHz frequency of the second reference clock signal. Other sizes of counters are possible based on the frequency of the synchronization signal and/or the second reference clock signal 335.

In some embodiments, the FPGA315-1 starts a counter in response to initiating the transfer of the common clock signal 335 from the TE0-W of the network element 100-1 to the TE0-E of the network element 100-0 using the link 220-0. At FPGA 315-0, common clock signal 335 is received as received common clock signal 340. Since network element 100-0 is configured to operate in a slave role, FPGA 315-0 is configured to loop back the received common clock signal 340 as a recurring clock signal 350 on link 220-0 (shown as loop 345). The delay calculation module 320-1 collects the value of the counter when the received cycle clock signal 355 is received by the FPGA 315-1.

The delay calculation module 320-1 is further configured to calculate a clock phase delay value using the average acquisition value of the counter. In some embodiments, collecting the value of the counter occurs as a first instance, and the delay calculation module 320-1 is further configured to perform collecting at least a second instance of the counter value in response to receiving the looped back common clock signal from the network element 100-0. In this manner, the delay calculation module 320-1 may be configured to collect a predefined number of values of a counter to calculate a clock phase delay value corresponding to the link 220-0. In one example using five acquisition values for the counter, the clock phase delay value may be calculated according to the following equation:

where A0, A1, … A4 represent five collected values of the counter and 200MHz represents the frequency of the second reference clock signal generated by the FPGA 315-1. The various delays included in the paths (e.g., delays in FPGAs 315-0, 315-1, backplane (backplane) routing delays, component propagation delays, and routing delays) may be reflected in the calculated clock phase delay values.

In general, the accuracy of the phase delay value measurement depends on the frequency at which the counter operates. In this case, the phase delay value can be calculated with an accuracy of ± 5 nanoseconds (ns) for a clock frequency of 200 MHz. The accuracy of the phase delay value measurement may also be improved when the contributions from the delays of the transmit path and the receive path are relatively balanced. To accomplish this, the logic 305-0, 305-1 may use relatively simple logic elements in the FPGAs 315-0, 315-1 and buffers and/or RS422 transceivers, which may advantageously avoid delays associated with packet processing.

Fig. 4 is a diagram 400 illustrating delay compensation between two network elements according to embodiments described herein. The process illustrated by diagram 400 may be used in conjunction with other embodiments disclosed herein, for example, between adjacent network elements 100 of the communication system 200 illustrated in fig. 2.

In diagram 400, FPGA 315-0 includes a delay computation module 320-0 and a delay compensation module 325-0. Delay compensation module 325-0 is configured to implement clock delay compensation(s) for TE port(s) of network element 100-0 (i.e., having a slave role).

The FPGA315-1 is configured to transmit the common clock signal 405 from the TE port TE0-W of the network element 100-1 to the TE port TE0-E of the network element 100-0 using the link 220-0. The first DPLL 310-0 of logic 305-0 is configured to receive the common clock signal 405 as a received common clock signal 410 and generate a first reference clock signal frequency synchronized to the received common clock signal 410.

In one non-limiting example, the first DPLL 310-0 generates a first reference clock signal having a frequency of 25 MHz. FPGA 315-0 is also configured to receive a first reference clock signal and internally generate a second reference clock signal having a frequency greater than the first reference clock signal. In one non-limiting example, the second reference clock signal has a frequency of 200 MHz.

In some embodiments, FPGA 315-0 includes two counters that receive a second reference clock signal. For example, each of the two counters may comprise a 32-bit timestamp counter corresponding to a 200MHz frequency of the second reference clock signal. Other sizes of counters may also be used based on the frequency of the received common clock signal 410 and/or the second reference clock signal.

In some embodiments, FPGA 315-0 is configured to set each of the two counters with a predefined value and to decrement each counter in response to a different attribute of the received common clock signal 410. In one embodiment, the predefined value includes a first period corresponding to the frequency of the received common clock signal 410 divided by a second period corresponding to the frequency of the second reference clock signal. For example, each counter may be set to a value of [0x61A8-0x (phase delay/5) ], where 0x61A8 corresponds to a hexadecimal value (125us/5 ns). The 125us value is a first period corresponding to the compensated 8kHZ common clock signal and the 5ns value is a second period corresponding to the 200MHz frequency of the second reference clock signal.

FPGA 315-0 is configured to decrement a first counter of the two counters from a predefined value in response to a positive edge of the received common clock signal 410 and decrement a second counter of the two counters from the predefined value in response to a negative edge of the received common clock signal 410. When the first counter is decremented to a zero value, FPGA 315-0 generates a positive edge of phase compensated clock signal 415. When the second counter is decremented to a zero value, FPGA 315-0 generates a negative edge of phase compensated clock signal 415. In response to decrementing to a zero value, FPGA 315-0 resets each of the two counters to a predefined value and repeats the decrementing process to generate subsequent transitions of phase compensated clock signal 415. Phase compensated clock signal 415 is sent from FPGA 315-0 to the second DPLL of logic 305-0 and network element 100-0 is synchronized with phase compensated clock signal 415.

Advantageously, the implementation shown in diagram 400 is relatively simple and does not require addressing packet protocols or the use of complex software stacks. Despite being so simple, this implementation can achieve high accuracy (e.g., within 10 ns). This implementation may use only minimal additional hardware logic, which reduces cost and power consumption relative to alternative implementations. For example, using the precision time protocol (IEEE 1588) or Global Positioning System (GPS) time to achieve clock synchronization with comparable accuracy is relatively complex and expensive.

Fig. 5 is an exemplary method 500 for configuring a communication system including multiple network elements according to embodiments described herein. The method 500 may be used in conjunction with other embodiments disclosed herein, for example, between adjacent network elements 100 of the communication system 200 shown in fig. 2. The logic for performing the method 500 may be implemented in one or more network elements and/or a separate computing device communicatively coupled to one or more network elements.

The method 500 begins at block 505 where the logic receives a reset signal. In some embodiments, each of the plurality of network elements is transitioned to an initialization mode in response to a reset signal.

At block 515, the logic determines whether more than one network element is detected in the communication system. If only one network element is detected ("no"), the method 500 returns to block 515. If more than one network element is detected ("yes"), the method 500 proceeds to block 525 and the logic transitions the detected network element to a predefined listening mode. In some embodiments, each port of each network element is converted into a predefined listening mode in which a received common clock signal is looped back.

At block 535, the first network element learns one or more neighboring network elements and calculates a phase delay value for each of the one or more neighboring network elements. In some embodiments, the first network element transitions to a predefined learning mode while the other network elements remain in a listening mode. In some embodiments, block 535 may be performed in conjunction with fig. 3. At block 545, the first network element returns to the listening mode after completing the learning of the plurality of ports of the first network element. Blocks 535 and 545 may be repeated for each remaining network element.

At block 555, the logic determines whether one or more neighboring network elements have completed learning. At block 565, the logic assigns a role to each network element. In some embodiments, one of the network elements is assigned as a clock master and the remaining network elements are assigned as clock slaves or isolated.

In some embodiments, the role of each network element is assigned using one or more predefined criteria. In one embodiment, the quality values of the clock signals generated by each network element are compared. The network element with the largest quality value may be assigned a clock master role. In another embodiment, the first network element to start up first is assigned a clock master role. In some cases, the network element that is first activated may be selected when the quality value is not determined or when the quality value does not sufficiently distinguish between different network elements. Other predefined criteria are also possible.

At block 575, the logic assigns a role for each port of each network element. Some non-limiting examples of roles include a "primary" role, a "backup" primary role, an "alternate" primary role, a "forward" role, and a "disable" role. The method 500 ends after completing block 575.

Fig. 6 is an exemplary method 600 for configuring a network element within a communication system according to embodiments described herein. Method 600 may be used in conjunction with other embodiments, for example, to configure network element 100 of fig. 1 as part of method 500 of fig. 5. In some embodiments, the network element includes a plurality of ports.

The method 600 begins at block 605 where the network element converts the plurality of ports to a predefined listening mode. At block 615, the network element determines whether the first port can learn a neighboring network element. In some embodiments, the network element converts the first port into a predefined learning mode while the remaining ports remain in a listening mode. If the first port cannot learn the neighboring network element ("NO"), the network element sets the first port to the disabled mode at block 625. If the first port does learn of the neighboring network element ("yes"), the method 600 proceeds to block 635 and the network element calculates a phase delay value associated with the neighboring network element. Blocks 615, 625, and/or 635 may be repeated for each remaining port of the network element.

At block 645, the network element switches the plurality of ports to a listening mode. At block 655, the network element assigns a role to each port based on the roles assigned to the network element. In one non-limiting example, each non-disabled port may be assigned a "forwarding" role for network elements that are assigned a clock master role to enable a common clock signal to be sent from a port of the network element. The assignment of port roles will be discussed further with respect to fig. 7-10. Method 600 ends after completion of block 655.

Fig. 7 is a diagram 700 illustrating synchronization states of a communication system according to embodiments described herein. The implementation shown in diagram 700 may be used in conjunction with other embodiments, for example, using communication system 200 shown in fig. 2. In some embodiments, the synchronization status is determined after the learning process is completed by the plurality of network elements, in accordance with method 500 (fig. 5) and/or method 600 (fig. 6).

In diagram 700, network element 100-0 is assigned a clock master role and network elements 100-1, 100-2, and 100-3 are assigned clock slave roles. In some embodiments, network element 100-0 switches each of its TE ports to a "forward" role (F) to transmit a common clock signal when assigning the clock master role.

For each of the clock slave distributed network elements 100-1, 100-2, 100-3, the first TE port at which the common clock signal is detected is assigned a "master" (M) role, and clock recovery is enabled on the first TE port. As shown, the TE port TE0-W of network element 100-1 and the TE port TE0-E of network element 100-3 are switched to primary roles.

At the clock slave distributed network elements 100-1, 100-2, 100-3, the second TE port is assigned a "backup master" (B) role when the second TE port receives the common clock signal after the first TE port is assigned the master role. As shown, TE port TE1-W of network element 100-1 and TE port TE1-E of network element 100-3 are switched to a backup primary role.

On the clock slave distributed network elements 100-1, 100-2, 100-3, the two TE ports opposite the first TE port (assigned the master role) are assigned the "forwarding" role. As shown, TE ports TE0-E and TE1-E of network element 100-1 are switched into a forwarding role. For network element 100-2, TE port TE0-W receives the common clock signal from (forwarding) TE port TE0-E of network element 100-1 on link 220-1, and TE0-W of network element 100-2 is switched to the master role. TE port TE1-W of network element 100-2 is switched to a backup role.

Since the TE ports TE0-E and TE1-E of the network element 100-2, and TE0-W and TE1-W of the network element 100-3 are located opposite the primary role TE ports of the network elements 100-2, 100-3, they will typically be assigned a forwarding role. However, in addition to forwarding the common clock signal, each of these TE ports also receives the common clock signal, and each of these TE ports is assigned the "alternate master" (a) role.

Fig. 8-10 each illustrate an updated synchronization status of a communication system according to embodiments described herein. In each of these figures, the updated synchronization status may reflect a dynamic reconfiguration of the communication system in response to a link degradation or failure.

FIG. 8 is a diagram 800 of a single link 220-1 failing (indicated by a break 805). Both the TE port TE0-W of network element 100-2 (previously assigned the primary role) and the TE port TE0-E of network element 100-1 (previously assigned the forwarding role) are assigned the "disable" (D) role. As TE port TE0-W is lost as the primary role of network element 100-2, TE port TE1-W of network element 100-2 switches from the backup primary role to the primary role, thereby enabling synchronization and delay compensation on TE port TE 1-W.

FIG. 9 is a diagram 900 of multiple links 215-1, 220-1 failing (represented by respective interrupts 905, 910). The TE port TE0-W, TE1-W of the network element 100-2 and the TE port TE0-E, TE1-E of the network element 100-1 are each assigned a disabling role.

Since both the primary role and the backup primary role of network element 100-2 are lost with multiple link failures, TE port TE0-E, TE1-E, which has an alternate primary role, is switched to both the primary role and the backup primary role. As shown, TE port TE0-E is assigned a primary role and TE port TE1-E is assigned a backup primary role. Further, since the TE port TE0-W, TE1-W of the network element 100-3 no longer receives a common clock signal from the network element 100-2, the TE port TE0-W, TE1-W transitions from the backup primary role to the forwarding role.

FIG. 10 is a diagram 1000 of multiple links 215-0, 220-0 failing (represented by respective interrupts 1005, 1010). The TE port TE0-W, TE1-W of the network element 100-1 and the TE port TE0-E, TE1-E of the network element 100-0 are assigned a disable role.

The TE port TE0-W, TE1-W of the network element 100-3 is switched from the alternate master role to the forwarding role. The TE ports TE0-E, TE1-E of network element 100-2 transition from the alternate primary role to the primary and backup primary roles, respectively. TE ports TE0-W, TE1-W of network element 100-2, which are disposed opposite the assigned primary role and the backup primary role, are switched to the forwarding role. The TE port TE0-E, TE1-E of network element 100-1 switches from the forwarding role to the primary role and the backup primary role.

Although not shown, if a link connected to a TE port of a particular network element 100 having primary and backup primary roles fails and no alternate primary role is assigned to the TE port within the communication system, a "waiting" role is assigned to any forwarding port of the network element 100 because the network element 100 does not receive a common clock signal for forwarding to neighboring network elements. Once the common clock signal is again received by the network element 100, the TE port in the waiting role may switch to the forwarding role.

In the foregoing, reference has been made to the embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to the specifically described embodiments. Alternatively, any combination of the described features and elements, whether related to different embodiments or not, is contemplated to implement and practice the contemplated embodiments. Moreover, although embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the scope of the disclosure. Accordingly, the foregoing aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s).

As will be appreciated by one skilled in the art, the embodiments disclosed herein may be embodied as a system, method or computer program product. Accordingly, aspects may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," module "or" system. Furthermore, aspects may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied in the media.

Any combination of one or more computer-readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium is any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

Aspects of the present disclosure are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments presented in the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In view of the foregoing, the scope of the present disclosure is to be determined by the claims that follow.