SCinet & Cisco Are Training the Next Generation of NetOps Engineers • SC23

September 12, 2023

SCinet

With only two months to go, final preparations are underway for the installation of the SC conference network, also known as SCinet. Run entirely by volunteers, months of meticulous planning will soon be realized with the start of the network deployment process, referred to as the “staging” and “setup” weeks. While many teams make up the SCinet group, all are required to be nimble and adapt to a constantly changing environment. However, for the Routing Team at SCinet, embracing constant change and pushing the envelope is not only a goal, it’s a necessity.

SCinet Diamond Contributor

SCinet Routing & Cisco NSO

Over the years, the Routing Team at SCinet has piloted different automation solutions to enhance their collective ability to scale and accomplish more in less time. For SC23, the Routing Team has partnered with Cisco to use the NSO orchestrator application. This app helps standardize configurations, simplify the deployment of new devices, and minimize human error.

Corey Eichelberger

SCinet Routing Team Co-Chair

“By leveraging the programmatic interfaces of modern network hardware and automation technologies, the Routing Team can realize efficiencies in how we build and manage the SCinet network,” explained Corey Eichelberger, SCinet Routing Team Co-Chair from the Texas Advanced Computing Center (TACC). “This allows the Routing Team to free up valuable time and resources for our volunteers to work on the more experimental projects that SCinet is known for.”

Kalina Dunn

SCinet Routing Deputy Chair

There is also a skills gap across the industry for network engineers who understand how to write software. Routing Team Deputy Chair Kalina Dunn, from GlobalNOC at Indiana University, explained: “Using technologies like Cisco NSO, we are also looking to provide our volunteers with NetOps experience and training, which are typically hard to come by in the networking industry. By letting them learn and experience how these technologies work, they can bring their experience back to help solve similar challenges their home institutions are facing.”

Cisco NSO Architecture

Cisco NSO is a multi-vendor orchestration platform that provides a single programmatic API (application programming interface) to the network [1].

“Cisco NSO can provide turnkey solutions to automate almost any hardware with an IP address and login, providing the HPC community a multi-vendor tool that will manage different vendors’ nuances of managing devices via the Cisco NED (Network Element Driver),” said Patrick Finnegan, a Network & Security Architect at Cisco and SCinet Routing Team member.

Combining various technologies to address the permutations of vendor equipment is a longstanding challenge for SCinet. Routing Team member Bartosz Drogosiewicz, from ICM University of Warsaw, offered his take on the challenges posed by the multi-vendor issue.

“The industry is at this weird but perfectly understandable point where every vendor tries to have its own automation solution that is only compatible with their own products,” Drogosiewicz said. “If you’re working in a multi-vendor environment, it becomes a challenge to orchestrate all the different devices and services. There’s a lot of little things and quirks that pile up while discovering nooks and crannies of old and new devices. That becomes the hard part: the little things that need to be documented and passed on.”

Bartosz Drogosiewicz

SCinet Routing Team Member

Patrick Finnegan

Cisco Network & Security Architect

Bringing Automation Home

Industry experts offered their suggestions for beginning automation:

Finnegan: “Start small and involve everyone in the organization. Don’t try to boil the ocean. Writing down multiple pain points and then voting on which parts to automate while keeping everyone involved is key.”

Dunn: “Automate things that will change the most and be the most impacted by human error first. These are usually the most time-consuming without automation.”

Eichelberger: “Decide on your source of truth. Once that has been decided, start small and look for small types of changes that could make a big impact.”

Drogosiewicz: “Talk with fellow admins at your institution and others. Exchange experiences and concerns. Sometimes you spend a month of research and development only to find out you really don’t want to do something a certain way.”

An Automation Journey

The SCinet Routing Team has long been a proponent of technologies to simplify management and provide more intelligent networks. While scripting against network elements is a concept that has been applied for decades, the Routing Team began adopting more robust techniques, such as Software Defined Networking (SDN) and OpenFlow, as far back as SC15.

OpenFlow, OpenDaylight, and Ryu (SC15–SC17)

OpenFlow was a transformative step toward adding more intelligence and programmability to the network’s dataplane. This provides more granular control over traffic flows in the network. OpenFlow was first introduced by the Open Networking Foundation in February 2011 [2].

The SCinet Routing Team leveraged the multi-vendor aspect of the SCinet network to test the robustness of equipment support for the OpenFlow protocol. At SC15, the Routing Team deployed the OpenDaylight OpenFlow controller. Network service that once required individual engineers logging into many devices, OpenDaylight could deploy between the core network and the booth at the push of a button [3]. This experiment was continued as part of SC17 using the Ryu SDN controller [4].

Ansible/NAPALM and Faucet (SC18)

Despite OpenFlow being adopted in Research and Education networks, industry support for OpenFlow among hardware vendors declined [5]. The Routing Team began to investigate alternative open-source projects being leveraged for server automation and orchestration.

During this period, the SCinet network was split between two different automation/orchestration frameworks. The first half of the network services were deployed using a combination of Ansible and NAPALM. The NAPALM library provided a multi-vendor unified API for programmatic network equipment access. The Ansible application provided an orchestration engine and the ability to automate workflows [6].

A model of configuration intent was exported from the SCinet Database (called Intranet). The final piece of this solution was the Flask web container application developed by the Routing Team, which acted as a web user interface front end for Ansible/NAPALM and the SCinet Intranet export script called GenConfig.

The other half of network services were deployed using an OpenFlow controller called Faucet. Despite OpenFlow adoption being on the decline, Faucet provided a unique solution as vendor OpenFlow support must be certified as compatible by the Faucet Foundation. This approach helped ensure consistency of OpenFlow support between hardware vendors.

Either the hardware is compliant with the required OpenFlow features, or it is not. Faucet also separates the SDN controller function from the operation of the network devices. An operator could update the software on the Faucet controller, and the network will continue operating using its current known good configuration [7].

Cisco NSO (SC19–SC23)

Despite previous success, the multi-vendor aspect of SCinet continued to present challenges for managing configurations and introducing new contributors in the network. Groundwork began in SC19 to build a virtual SCinet R&D network using the GNS3 hypervisor, allowing multi-user topology management. This allowed the Routing Team to expand their R&D cycle for SCinet from a few weeks to year-round. Using network design data from previous SCinet networks, the Routing Team leveraged this environment to pilot configuration management and orchestration with NSO.

Find Your Own “Route” with SCinet

In addition to research and education networks, getting hands-on experience in a multi-vendor network and learning from the brightest minds across educational institutions, government agencies, and HPC sites are among the many reasons volunteers are drawn to SCinet. Drogosiewicz offered his own reasoning about why others may want to join.

“SCinet builds something totally unique,” he added. “I call it ‘The Woodstock of IT.’ It’s a three-day festival with the world’s fastest temporary network built and disassembled within a week. Being a part of the Routing Team means that you’re doing all the different aspects of architecture, design, R&D, configuring, and troubleshooting. Everyone participates in every aspect of it, and it’s a unique chance to broaden your knowledge.”