Nikola Mihaylov, one of Systal’s Principal Network Architects, explains how out-of-band networks provide a vital service in maintaining business continuity and protecting customer reputation.
… Midnight and the on-call phone rings. The Network Operations Centre (NOC) team is reporting a high-severity incident in a customer environment – nothing is accessible. The only way to access the data centre and resolve the issue is to have hands and eyes support on-site. The on-site engineer is up to 4 hours away and the customer is in panic mode as the outage is heavily impacting their business. It’s going to be a difficult period for both the customer and the managed service provider: penalties may be paid, business will suffer, and reputational damage is likely.
This all sounds dramatic, but unfortunately, this is a very real scenario which most engineers with experience in service integrator fields can attest to.
How to avoid this situation and minimise customer impact:
For various reasons, out-of-band management is often a neglected area in most customer data centre designs. However, no matter how much one would like to think that having redundancy and backups in place natively should be enough to allow fast recovery in various outages or security incidents, the implication of specific failure conditions has a high penalty. This is especially true in remote-managed environments where support personnel need to have the ability to react quickly and swiftly to resolve network or server issues with a guaranteed access mechanism. This is where out-of-band management can help.
Out-of-band management, what is it?
First, let’s review what the options are for managing the customer environment. This is not only limited to network devices, but also to the various appliances and servers one can have in their data centre, campus, and even remote offices.
- In-band management – This method allows the managed service provider to manage the customer environment by using the same data path as the production traffic, only having the management traffic logically separated from the production traffic. Yes, this is a cost-effective method, but as one can see, in case of production issues, it will be very easy to also lose management access to the managed infrastructure (fate sharing). In this situation, when there is an outage, often the only possible solution is to dispatch an engineer on-site for further troubleshooting and restoring the service. Often this is an expensive and time-consuming activity that impacts customer business.
- Out-of-band management – This is a separate, parallel network that allows independent management access to the infrastructure, not using the production network (not fate sharing). Out-of-band will allow access to network devices and servers even if there are serious issues in the main production network like provider outages, security incidents, or other similar events.
Elements of out-of-band management:
Today, most appliances and servers have dedicated, purpose-built management interfaces to provide out-of-band. Usually, these interfaces are independent of the data plane and only provide access to the management plane of the device. Last, but not least, most of the servers that are shipped on the market also have some type of out-of-band management tool or baseboard management controller, that will allow system provisioning, server management and monitoring tools.
Let’s discuss how out-of-band should be built from a network point of view:
- Independent Network: As we already clarified, out-of-band should be an independent network and it should consist of a separate set of switches, to which all infrastructure management ports will be connected.
- Independent Access: Next, out-of-band should have independent access to external networks, that are not dependent on production. Security for the out-of-band network is of paramount importance and access to this network should have a similar security policy, if not even stricter, than the one for the production environment.
- Dedicated Terminal Server: Most of the network devices and servers used in production environments also have a legacy console port (RS-232) which is used for the initial setup of those devices. However, these interfaces also provide alternative methods to access and manage devices in the customer infrastructure. To centralise this management and allow access to multiple devices via a console port, a dedicated terminal server can be introduced into the out-of-band infrastructure. Terminal servers can also provide an additional layer to allow access to the network infrastructure in events where main access to the out-of-band network is not possible, by using Dial-up or LTE cellular access, which is activated once the main access is not available.
- Remote Power Switches: Remote power switches are another type of network management device that can often be used together with the terminal servers to remotely power equipment in the customer data centre.
Why do we really need out-of-band?
- Constant access to managed infrastructure – Out-of-band networks can help to ensure business continuity and help remote administrators access the customer environment via an alternative path.
- Isolated management infrastructure – Out-of-band management can provide an additional layer of security and the customer environment can maintain resistance against multiple security threats.
- Reduce downtime in case of issues – By providing separate access to the management of devices (console and network), out-of-band can provide a faster way to access devices for troubleshooting, upgrades, and other activities.
- Power-up/down and restart of managed devices – With the introduction of remote power switches in the data centre or critical location, engineers can quickly restart a troublesome device or even power it down to isolate it from the rest of the environment. This can be especially critical in the case of security threats to the organisation.
High-level build to secure access to the out-of-band network:
The following picture shows a high-level example diagram for a build to an out-of-band network to the customer premise (the design may change based on specific customer security policies).
In short, most customer IT infrastructure is located far away from support personnel, and it requires remote management. Let’s consider the following scenario. Command centre infrastructure (VDIs, management and monitoring servers) is in one of the cloud providers, from where support engineers are using VDI or other kinds of jump servers to reach the customer environment. This cloud environment has a secure connection to the customer’s IT infrastructure, such as a dedicated private network connection to the on-premises location or SD-WAN / IPSEC VPN which is terminated on an on-premises router/firewall to ensure that connectivity to the out-of-band network is secure. This connectivity is considered primary and should be built with appropriate resilience.
On the customer premise, a separate management LAN infrastructure should be built and will be used for management connectivity of IT infrastructure. To the same management LAN infrastructure, a terminal server(s) is connected to provide access to devices with console access. This terminal server should have the option to establish a secondary connection to the network management infrastructure in case the primary link is down (Dial-up or LTE cellular modem, or secondary internet link with VPN connectivity). Secondary console connectivity to the infrastructure provides a fail-safe mechanism to access the infrastructure in the event of multiple failures.
How can Systal help you create a secure out-of-band management solution?
Systal partner with leading manufacturers of out-of-band equipment to successfully deliver an infrastructure for several financial and manufacturing customers, ensuring that in the case of serious issues, we can help their business recover quickly.
Working with our customers’ network and security teams, we can provide them with a secure and reliable out-of-band solution, and peace of mind that their IT infrastructure is secure.
Conclusion
Although at first sight out-of-band networks appear to be unnecessary and costly, they provide a vital service in maintaining business continuity and protecting customer reputation in the event of a failure condition. Nobody wants that call at midnight or to have an incident played out by the national press, so it’s worth viewing out-of-band as an investment as opposed to a cost. Well-designed out-of-band infrastructure can provide businesses with a secure way to have their network managed by remote personnel and minimise the cost and time for onsite support to reach an impacted location when response time is critical.
Nikola Mihaylov is a Principal Network Architect at Systal Technology Solutions, designing network solutions for some major customers in the financial, retail, manufacturing, and utility sectors. Nikola has over 15 years of experience in the networking field, managing, designing, and consulting customers to offer fast and agile solutions to meet their business needs. He is one of Systal’s key technical leadership resources, leading LAB in Brno and helping Systal provide data centres and network security solutions.
Contact Systal's Experts