Fall has arrived, and the SCinet team has begun its takeover of the Colorado Convention Center to start constructing the SCinet network for the SC23 conference. Volunteers of SCinet are among the earliest to arrive in Denver for the event called “Staging” week. At Staging, SCinet volunteers begin installing equipment borrowed through the SCinet contributors program under a tight deadline to enable basic network connectivity that will support the SC conference. However, for SC23, SCinet will not simply construct an ordinary network in Denver. This year’s version will provide an astonishing 6.71 Tbps (terabits per second) of network bandwidth to the convention center. This speed is sufficient to download the Library of Congress’ complete 21 petabyte (equivalent to 21000 terabytes) catalog in just over seven hours¹.

Achieving this goal necessitates forging robust partnerships with a multitude of external stakeholders, encompassing Research and Education (R&E) networks, commercial network providers, and researchers spanning the globe.

Wide Area Networks Need Wide Social Networks

Carlos Rojas-Torres

SCinet WAN Team Member

Leadership & Teamwork

The existence of the SCinet wide area network, or WAN, and collaborative efforts of SCinet contributors are absolutely essential for the realization of the conference network.

“The WAN team requires a strong and committed leadership team. The preparation and coordination of contributors, volunteers, and the SCinet committee is a multi-year endeavor,” explained Carlos Rojas-Torres, SCinet WAN team member from the University Corporation for Atmospheric Research (UCAR) and Front Range GigaPoP. “The team is composed of some of the most talented people in the industry, and it’s practically impossible to walk away from there without learning something new every single day.”

Relationships Matter

The WAN team’s ability to stay ahead of the numerous challenges they encounter each year can be attributed to the cultivation of these enduring multi-year partnerships and relationships. 

“The culture and relationships are what makes the WAN team successful,” said Chris Skaar, SCinet WAN team co-lead from University of Illinois Urbana-Champaign. “The leads for the WAN team have, over the years, fostered a culture of hard work, supporting one another, learning from each other, and developing long-lasting connections. We also try to help our contributors be as involved as possible and ensure that our team really learns about the products that they are donated.”

Chris Skaar

SCinet WAN Team Co-Chair

Big Tbps Requires Big Creativity

SCinet’s WAN design is intricately linked to the specific destinations researchers wish to connect to and the use cases that drive the presentation of their research at SC.

Cody Rotermund

SCinet WAN Team Deputy Chair

Putting the Puzzle Together

“Leading up to Staging week, it’s almost like we’re putting together a puzzle,” explained Cody Rotermund, SCinet WAN Team Deputy Chair from the Energy Sciences Network (ESnet). “We get some of the pieces through the end researchers or through the SCinet Network Research Exhibition (NRE) process. However, many of the puzzle pieces you need to form a complete picture aren’t really known until later in the year.  Essentially, to start putting an initial picture together, we do quite a bit of forecasting based in part on what we did in previous years. With that in mind, we’ll take a look at what fiber currently exists, and what we need to work with our contributors, to build out and go from there.”

A one-size-fits-all solution simply doesn’t exist. Each scenario demands a distinct technical approach and the judicious pairing of technology. The SCinet WAN leadership team invests a significant amount of time in assessing emerging technologies from WAN equipment contributors and aligning them with researcher use cases and connectivity requirements to meet the demanded network bandwidth.

“Once we have a handle on the initial fiber plant design, we then work with our hardware contributors and attempt to line up the best technical solution for the use case,” Rotermund added. “In some cases, a network carrier contributor might be able to provide end-to-end connectivity over their Dense Wavelength Division Multiplexing [DWDM] system. In other scenarios, we collaborate with our hardware contributors to connect into the carrier’s DWDM system using our own transponder modems.”

The World Is Our Stage: SCinet Global WAN Map

Finding the best technological solution often requires outside-the-box thinking. For example, Skaar noted that SCinet uses multiple contributors’ equipment to run “alien waves” across existing Line systems. An alien wave is when a third party uses their own transponder modems to transmit data over a carrier’s DWDM network. 

“This involves using Vendor A’s transponders across Vendor B’s line system and requires a lot of coordinated effort as many times these circuits must be manually configured on both vendors’ equipment,” Skaar described. “This year, we are utilizing ZR optics, which use tunable, optical frequencies in the same spectrum as DWDM to transmit data over shorter distances, under 10 km. This allows us to multiplex or ‘mux’ these wavelengths into our DWDM system from our local Point of Presence [POP] sites to the convention center, reducing the need for extra fiber.”

Testing, Testing, and More Testing

The performance tolerances for operating connections at speeds like 100 Gigabit Ethernet (GE), 400 GE, and beyond are exceptionally tight. The SCinet WAN team dedicates significant effort to collaborate with various network carriers and conducts thorough performance validation and testing throughout the Staging phase and in the days leading up to the SC23 conference. Time is of the essence during this critical period, and the WAN team’s innovative use of technology for remote testing has yielded significant benefits and results.

By employing a micro-electromechanical systems (MEMS) optical switch, the WAN team has the ability to implement dynamic fiber topology adjustments within the convention center, eliminating the need for manual fiber swapping in the field. This innovation enables them to programmatically establish connections between WAN circuits and test equipment. Once the connections have been validated, they can be seamlessly redirected to the appropriate router or switch, streamlining network operations and enhancing efficiency.

“This saves us valuable time of manually repatching fiber jumpers, as well as providing extra optical measurements, which is crucial in troubleshooting,” Skaar added.

Share Your World with Us

Are you seeking hands-on, experiential training in the construction and operation of a wide area network? Join SCinet and learn from some of the most brilliant minds in the field! The SCinet WAN Team includes optical engineers from diverse backgrounds, including R&E, government, industry, and HPC. As Rojas-Torres strongly encouraged, SCinet offers a valuable opportunity for knowledge and skill acquisition in the world of wide area networking. 

“Don’t hesitate: go for it!” he implored. “The friendships, professional networking, and exposure to new technologies you’ll experience while being part of the WAN team will stay with you forever.”

Learn more about how to participate in SCinet for SC24:

Participate in SCinet

More About WANs

What Is a WAN and What Makes It Work?

A Wide Area Network (WAN) encompasses a set of technologies and service provider networks designed to interconnect various Local Area Networks (LANs) over extensive distances. A LAN enables networked devices, e.g., computers, to communicate and share resources within a confined geographical area or under a single administrative entity, such as a school, data center, or business. If you think of the LAN as an isolated island, the WAN acts as the bridge that facilitates the exchange of data and traffic between these distinct islands.

To illustrate with a practical example, consider a scenario where a university located in New York operates its own LAN and seeks to establish a connection with a remote resource hosted in the Cloud, which resides within another LAN situated in Los Angeles. In this instance, the WAN plays a crucial role by facilitating long-distance communication between these two organizations. This communication is achieved using fiber-optic cables as the transmission medium and specialized Dense Wavelength Division Multiplexing (DWDM) equipment, which enables efficient multiplexing and transmission of data over various wavelengths, ensuring reliable connectivity between the two LANs separated by a significant geographical distance.

DWDM and Optical Networking

A device called a transponder modem takes one or multiple network Ethernet connections from a LAN and encodes each connection into a specific frequency range within the visible light spectrum, also referred to as a channel or wavelength. The DWDM equipment functions like a prism, taking these individual wavelengths generated by the transponder and amalgamating them into a composite signal for transmission over a single fiber. The bandwidth capacity of each of these wavelengths varies based on the optical properties of the fiber, such as distance, loss, and reflection. It is common to encounter DWDM equipment that can support up to 80 distinct wavelengths over a single pair of fibers. This capability allows the WAN provider network to pack a substantial amount of network bandwidth over a single fiber efficiently.

After the DWDM equipment transmits the composite signal onto the fiber, signal degradation becomes more pronounced the farther it has to travel. To counter this, amplification equipment is strategically placed along the fiber route to rejuvenate the signal. Eventually, it reaches the DWDM equipment at the opposite end, which then separates the composite signal into its original wavelengths. These wavelengths are received by the transponder modem, which converts them back into individual Ethernet connections for the LAN.

References

¹ https://www.loc.gov/programs/digital-collections-management/about-this-program/frequently-asked-questions/