Cisco Routed Optical Networking
Communication service provider (CSP) traffic continues to be predominantly IP services, and consequently:
Current CSP network infrastructure is not optimized for transport of IP services and is challenged with high TCO attributed to:
To address these pain points and build networks to scale effectively and efficiently to meet increasing traffic demands, Cisco is advocating a new routed optical networking architecture to drive network simplification and flexibility and reduce TCO network operations:
The Cisco Routed Optical Network architecture features:
Communication service providers (CSPs) are working to balance their need to expand network capacity while controlling costs in the face of flat revenues. Traffic will continue to grow from video, gaming, and virtual/augmented reality as well as from the promises of mobility with 5G and future technologies. CSPs need to continue meeting the demand for this new capacity, while also rolling out new revenue generating services and lowering their costs. Looking at forecasts, a CSP will need to move 11 times more traffic per $1 of infrastructure investment in 2022 than in 2012.
Figure 1. Cisco Visual Networking Index (VNI) vs CSP network investment
Internet traffic has seen a compounded annual growth rate of 30% or higher over the last five years because more devices are connected, and more content is being consumed1.
Technology continues to evolve following Moore’s Law, and we continue to push Shannon’s Limit. From operations we see a strong drive to leverage automation, Software-Defined Networking (SDN), telemetry, machine learning, and artificial intelligence to lower operational costs. From a network architecture perspective, we continue to see investments in individual network layers, from packet to optical transport network (OTN) to dense wavelength division multiplexing (DWDM), with each layer evolving around capacity and flexibility in a siloed manner. This contributes to the cost and complexity of the overall network infrastructure.
Current architectures and topologies have proven ineffective in addressing these pain points. CSPs need a new approach to simplify the network and include automation tools that will improve utilization, maximize traffic controls, and provide deeper network visibility and orchestration. This transformative approach must be taken to:
The objective of this paper is to provide details on these transformative network architectures and technologies that form part of the Cisco Routed Optical Networking solution to facilitate efficient network growth for IP services.
Traditional network architectures are comprised of networking layers that rely on line cards for service handoff between the layers. This kind of layered architecture is highly inefficient as it consumes too much line card capital expenditure (CapEx) resources and relies on manual operations for service handoff between the layers. Moreover, each networking layer has its own control and management planes associated with it, which operate independently from each other. This creates huge complexities in service assurance, fault correlation, path optimization in terms of network utilization, as well as network planning and optimization. These complexities represent challenges to CSPs’ aspirations towards achieving service driven end-to-end, closed-loop automation across the entire network infrastructure. The total cost of ownership (TCO)2 associated with this network architecture is prohibitively high and will not allow CSPs to scale their network to meet the capacity demands of IP services in a cost-efficient manner.
Figure 2. Traditional Network Architecture
Cisco is committed to addressing these challenges by simplifying key technologies that enable convergence of the IP and optical domains. Routed optical network simplification is aimed at removing complexities inherent in the current architectural designs and multiple layers to allow CSPs to leverage their assets more effectively.
The routed optical networking architecture framework is a key enabler for the realization of the Cisco Internet for the Future vision. This new approach transitions networks from the siloed infrastructure to a new architecture that relies on a single control plane based on IP/MPLS in a converged router-to-router IP and optical network. This drives significant simplification and cost savings. It addresses the complexities and redundant networking layers that present bottlenecks to scalability and enables end-to-end automation in the CSP network infrastructure through:
Figure 3. Cisco Routed Optical Network Architecture
This architectural design provides the opportunity for significant cost savings but such a transformative architectural change could present a challenge to the operational constructs of many CSPs and thus involve a phased introduction. CSPs would take different paths to reach the envisioned routed optical networking architecture, and the Cisco Customer Experience (CX) team can help. The CX team has developed network models that can outline different levels of potential cost savings based upon the level of transformation adopted. Additionally, the CX team operates the Routed Optical Networking Architecture Transformation Advisory service that will help CSPs plan, validate, and implement the solution so they can achieve the expected savings.
Solution components of the routed optical networking architecture can interoperate with the ROADM-based optical network architectures that CSPs have in their network infrastructure today. For example, the coherent DCO optics already supported on the current IP aggregation devices may be leveraged in ROADM-based infrastructure today. The routed optical networking architecture is defined by two initiatives which address key inefficiencies in the network architectures of today:
Figure 4. Routed Optical Network Architecture Transformative Solution Components
Services are key to any communication service provider’s business. Assuring that all services can be provisioned quickly for faster revenue and can ride over the IP network with no additional latency, with the assurance that those services are visible for management, are key to this new environment.
Fundamentally, CSPs were forced into a position of building layered networks rather than an ideal service-focused network. Multiple efforts are underway to shift away from this model by leveraging recent technological and organizational enhancements. Some examples are:
Figure 5. Routing Line Card Bandwidth Enabled by Optical Pluggable Modules
Architectural changes are required to bring about these benefits by reducing complexity, maximizing capacity, and avoiding inefficient networks. Routed optical networking allows CSPs to operate a network that has a converged optical and IP layer that increases visibility and control while reducing costs. The convergence of the optical and IP layers can eliminate the dedicated OTN layer that carries private line services, or providers can leave the OTN and Muxponder network in place and grow new services on the IP layer and cap the OTN/TDM already in place. Services need to get from ingress to egress, meet SLAs, and be as cost efficient as possible while reducing carbon footprint. By focusing on the service layer, we can leverage PLE plus 400G ZR/ZR+ optics and automation enhancements together to converge layers, simplify the network, and enhance the lifecycle management of the network.
Converging private line services onto the packet layer may raise some concern around service latency. The concern arises from running critical TDM services over a packet network which traditionally meant introducing latency and reducing reliability. However, packet networks are no longer slow, nor do they incur latency, thanks to the hardware-based packet forwarding engines used in today’s networks. Latency can be controlled by using strict bandwidth accounting (RSVP-TE) and/or quality of service (QoS) mapping to strict priorities, ensuring no additional latency is added to the network.
To address the current pain points faced by CSPs, we leverage the following principles as guidelines for building the next-generation networks:
Routed Optical Networking architecture takes advantage of the technology advancements and new capacities being made available to deliver a network optimized around the service. It enables true network capacity optimizations while providing network simplification. This is accomplished by the collapsing of layers with reduced port-density trade-off optics and eliminating the complexity and redundant nature of layers that add unnecessary switching complexity to a network.
A fundamental requirement of optical bypass featured in both hollow core (HC) and optimal bypass (OB) architectures is that it requires leveraging ROADM switching elements in the optical network infrastructure to establish direct routes between routing devices (e.g. direct provider edge (PE) device to provider (P) device) in order to avoid H2H optical connectivity between adjacent routing devices (e.g. PE to PE). This optical bypass feature leverages optical transit on ROADM elements as well as high performance coherent DSP. This ensures optical reach performance to establish feasibility for optical connections between routing devices over challenging (due to distance or quality or both) long-haul fiber infrastructure. This means the HC architecture is much more reliant on leveraging the most powerful coherent DSPs to bridge the gap between Moore’s Law and Shannon’s Limit. As many CSPs are aware, this trade-off leads to low fiber utilization (monetization). This is due to lower channel count and high network costs attributed to large volumes of regeneration introduced as necessary measures for maintaining performance over large distances. Having large volumes of required regeneration defeats the primary purpose of the HC architecture, which is to minimalize the use of routing line cards as a transport function. In this case, the regeneration function is merely transferred from the routing device to the optical transponder line cards.
In contrast to the HC architecture, routed optical networking architecture seeks to avoid the challenges posed by the trade-off between channel capacity vs. performance characterized by Shannon’s Limit. It avoids the need for optical connectivity between routing devices over long distances by advocating optical connectivity only between adjacent routing devices.
Table 1. HC to OB to H2H Architecture Comparison
The Cisco Routed Optical Networking architecture takes advantage of advanced technology and features available today to build a new architecture that scales with technology, enabling a network that will:
With recent advances in packet switching silicon, Field Programmable Gate Array (FPGA) technology and stagnating innovation in the space of OTN framers, bit-transparent transport services are better delivered using emulation technology over a high capacity, scalable, and cost-effective packet network infrastructure. This solution, referred to as Private Line Emulation, allows bit-transparent point-to-point connections between a wide range of client ports which include OC-48, OC-192, 1GbE, 10GbE and 100GbE. It also allows for transparent Optical channel Data Unit (ODU-k) connections for Optical channel Transport Unit (OTU-k) client ports.
The connections are established using pseudo-wires in accordance with the Pseudo-Wire Emulation (PWE3) architecture and are leveraging an MPLS or Segment Routing (SR) underlay. This architecture can leverage SR transport with enhancements for circuit-style services to realize flexibility, efficiency gains, and simplicity of SR. This flexibility allows for a single, common SR/MPLS switching layer for private line services and any other carrier ethernet or IP service offering, reducing networking layers, complexity, and cost.
Topology lndependent-loop free alternatives (TI-LFA) are used to provide fast reroute protection to services for failure scenarios including link, node, and local shared risk link group (SRLG) by leveraging the post-convergence path. TI-LFA relies on SR to build a protection mechanism based on proven IP-fast reroute (IP-FRR) concepts to extend sub-50ms protection to private line services, like the 1+1 protection schemes that are commonly reliant on being served from an OTN switching network layer.
There are two main concepts driving changes in the routed optical networking architecture:
The direct integration of Digital Coherent WDM interfaces in the Router eliminates the traditional manually intensive service hand-off across the demarcation between the optical transport and packet domains. The result is a single network infrastructure that can be planned, designed, implemented, and operated as a single entity.
Automation gives you the speed to execute operations with known repeatable quality, analytics for better decision making, and a way to synchronize business decisions with application processes. Speed gives the advantage of faster service provisioning for reduced time to revenue, faster decisions for proactive planning and problem avoidance, and increased responsiveness to protect the customer experience. Automation is truly about looking at the process; taking it to machine speed and executing it at a completely repeatable level and do it statefully, so that the outcome is exactly what is intended. The step to automate processes brings visibility, insights, and actions to the network in a fully autonomous and ideally closed-loop fashion. This is done with Cisco Crosswork Network Automation, a portfolio of tools to simplify and normalize repeatable operations.
Automation is essential to the routed optical networking solution. Reducing workloads, positively impacting network performance and the client experience, provides a competitive and differentiated network experience. Few competitors can deliver automation on a multi-layered network - controlling both IP and optical with specific domain controllers. Controllers with microservices essentially manage the specific domains. They have the topology, discovery, inventory, southbound alarming, and device profiles. And since it is a multi-domain world, there are multiple vendors in each networking domain that must be considered. Thus, you need a hierarchical controller to manage across those domains to bring visibility, insights, and actions aggregated at a higher level. The IP and optical controllers show a more detailed level of granularity of the network. The multi-layer HCO provides a service level view and service assurance as it looks across domains.
Figure 6. Architecture-enabled network automation
As these networks come together, with routed optical networking and coherent pluggables in the routers, there is the need to have a common way to operationalize this. This is partly why IPoWDM failed 20 years ago, not because of the technology, but because it’s about bringing the operations together in a way that allows optical and IP teams to work together to deliver a service, assure it, and visualize it for the customer. That’s what multilayer automation brings to the solution.
The multilayer controller (HCO) provides end-to-end optimization and visibility. It effectively delayers the network and manages the services with dedicated controllers by talking directly to both CNC (IP controller) and ONC (optical controller) and any third-party network. The multi-layer automation architecture provides visualization, connectivity, and event awareness. It is vendor agnostic and communicated through open and standard service models. It is a solution for routed optical networking as well as for routing and optical platforms outside the solution, and is an industry standard solution.
Here are the elements of Cisco’s Crosswork Automation:
Figure 7 shows how Cisco is bringing it all together.
Figure 7. HCO’s function is an abstract hierarchical controller to stitch IP and optical layers
In the past, direct termination of DWDM interfaces on the routing device to eliminate transponders required a coherent DSP to be implemented on the router line card, which occupies real estate on the line card. This resulted in reduced port density and capacity on the router line cards. With the new pluggable coherent optics that incorporate the coherent DSP on the optical pluggable modules instead of the host line card, the maximum router capacity can be maintained with little to no density tradeoffs.
A key pillar of routed optical networking is the integration of the coherent pluggable modules. As router port densities increase, the CapEx spend transitions from the line card ports to the pluggable optics. A key enabler for the scale envisioned for the Internet for the Future is the 400G line rate. 400G coherent optics leverage a quad small form-factor pluggable (QSFP) with standardized specifications, which allows interoperability for easier adoption and gains of scale. Previous implementations relied on proprietary digital signal processors (DSPs) on router line cards that were not interoperable or subject to standardization.
800G and beyond will be featured in the next generation routing line cards which can support up to 24.6Tbps on a single blade. To ensure full flexibility, these line cards will support both coherent and grey optical interfaces (or a combination thereof) in the form of QSFP-DD form factor. The use of QSFP-DD for both coherent and grey optics can be leveraged to limit any tradeoff in terms of port densities and IP fabric capacity commonly associated with coherent optics on routing platforms (previously referred to as IPoDWDM). This means that a specific line card required to host coherent DWDM optical interfaces will no longer be required. A universal line card which can be flexibly deployed to support the termination of coherent DWDM, or grey optical line interfaces, can be realized. While the maximum number of 400G ZR/ZR+ ports on a single coherent line card is limited by the optical power budget and characteristics associated with the line, the 400G ZR/ZR+ line card can still support significantly higher port densities than what is feasible on the previous generation CFP2-DCO-based coherent DWDM line cards.
QSFP form-factor has been widely leveraged in the industry and Cisco has been a key contributor in promoting QSFP-DD multiple supplier agreement (MSA) through the Optical Internetworking Forum (OIF) as well as other standards organizations. QSFP-DD is strategic in the realization of 400GbE and specifically for 400G ZR+. It can dissipate the heat associated with the ZR+ interface as it incorporates the coherent transmission required to establish midhaul optical reach at the 400G line rate.
In the IP aggregation application space, IP/optical integration can already be served by leveraging IP aggregation devices which feature Modular Port Adapters (MPA) and Digital Coherent Optical (DCO) pluggable modules. The integration of DCO modules allows the direct termination of coherent DWDM interfaces directly into the IP aggregation devices without incurring the cost and complexity of optical transponder line cards. While the integration of CFP2-DCO modules into routing devices has already been leveraged to address key networking pain points in the IP aggregation space, the port densities associated with CFP2-based optics are too low compared with the QSFP pluggables employed in their grey optical line card counterparts to offer the similar TCO savings. This is primarily because in the IP aggregation space, typically a small number of 100G/200G line interfaces are terminated at each aggregation site which does not lead to significant deloading of the IP fabric. However, in the case of core/edge applications, the volume of traffic is significantly higher and therefore core/edge aggregation devices leveraging current DCO line cards are more prone to IP fabric deloading arising from lowered port density.
Figure 8. Evolution of pluggable coherent optics
Environmental sustainability is top of mind at Cisco as we work to improve our impact on the planet. Our efforts began with designing massively scalable systems that provide more features in smaller footprints. Smaller devices allow CSPs to reduce their real estate, reducing power consumption and leading to a lower carbon footprint. Cisco is also focused on recycling and reusing components and materials from our older systems to be less reliant on precious resources, and we are working through our supply chain to reduce packaging and integrate more sustainability sourced materials.
Convergence of the IP and optical layers further reduces the number of devices, positively impacting power consumption and reducing truck rolls. Network modeling of the Cisco Routed Optical Networking solution benefits shows up to 45% power reduction and up to 70% real estate savings. The enhanced programmability of the network through automation and new packaging of optics and other platforms to use less material can help our customers lower carbon dioxide (CO2) emissions and achieve their overall sustainability goals.
1 Cisco Annual Internet Report (2018–2023) White Paper
2 Cost considerations commonly associated with the Capital Expenditures (CapEx) as well as the Operating Expenses (OpEx) of owning and operating the network