SDN and Big Data - A Powerful One-Two Punch

SDN is fundamentally about programmability, the separation of forwarding from control, and the ability to dynamically configure service and network elements in real time. However, to reap the full benefits of SDN, SDN applications must dynamically respond to the environment in which they live. Big data offers the ability to characterize that environment, in terms of state, behavior, anomalies, and long-term trends.

While big data offers the promise of new and potentially surprising forms of insight, you don’t get this ‘for free’ by simply deploying Hadoop. You need to apply machine learning or statistical analysis techniques and non-trivial data science, based on the application context.  As analysis and visualization tools mature, the value of this insight will take an increasingly important role in both the design and long term operational aspects of converged data centers and in the design and running of services offered to users.

In the context of SDN, this convergence with big data offers a particularly interesting dimension - a feedback loop - the ability to take analytics and state change events and dynamically inject those back into SDN decision-making, guided by a policy engine. This promises to make next generation data centers highly tunable, highly automated and highly responsive to real-time changes in operational use and behavioral trends.

Imagine the capability to setup, teardown, and scale-out ‘service-chains’ for individual workflows dynamically, by time of day, geo-location, traffic demand, and security context; all under fine-grained policies and with different SLAs. Over time, this insight could be essential in maximizing top line, improving customer retention, raising brand awareness, and maintaining competitive edge. Although the gains from these insights are relatively modest, in today’s highly competitive market, it could be the difference between a market leader and an ‘also-ran’.

Challenges

While there is tremendous interest in both SDN and Big Data, to date, there are no clear standards or reference architectures detailing how these technologies are to be coupled, and no consistent data on efficacy in terms of TCO and ROI, particularly across specific vertical applications. Aside from greenfield sites, we are likely to see networks evolve through a mix of hybrid infrastructures for some years to come.

One initial challenge is enabling all vendor technologies to be agnostic, both in terms of SDN controller integration and the instrumentation itself. That being said, considerable progress is being made through initiatives such as OpenStack and OpenFlow. There is also a clear push by vendors to provide open APIs based on RESTful principles - key for data center orchestration and automation.

Another major challenge has to do with securing these big data lakes from external hacking and ensuring data integrity is maintained over time. Given the level of centralized control, security is a concern for SDN as well. Here again, there are several initiatives as well as emerging technologies such as block chain that will help when paired with traditional security and fine-grained security policies.

There are further challenges around instrumentation, storage and analytics - if we want analytics over real-time as well as long-term data then the granularity of the instrumentation needs to be considered.

This is only the ‘end of the beginning’

Bringing all of the potential from big data to SDN is not trivial; this field is evolving rapidly but there is still a way to go before we see complete and widely published reference architectures. Right now we are probably closer to the ‘end of the beginning’ for machine learning and analytics on big data, and the ability to inform SDN decision-making consistently and accurately, based on policy and dynamic feedback. Early adopters and more forward-thinking organizations are already gaining experience in the trenches, but many of the tools, interfaces and standards are still evolving. Furthermore, the closely related technologies such as virtualization, cloud, service-chaining, NFV, storage, and security are all in a state of flux. The next few years promise to be exciting as these areas converge towards fully programmable and responsive service architectures.

It is clear that the marriage of big data analytics and soft-programmable infrastructure will promote the ability for organizations to deploy services and support users in a way that was simply not possible in the past, either because it was too resource-intensive, too costly, or simply too risky to attempt. We should expect to see fully automated data centers using closed feedback of policy-driven programmable elements, instrumentation, and analytics become the norm.

The blurring between hard and virtualized infrastructures will continue, with hardware needed to enable core and edge scalability and for high performance low latency applications; but importantly - everything instrumented and accessible through common APIs and policy. Cloud environments in particular place huge strain on resources and there will be a continued tension between vertical and horizontal scaling, between hard and virtualized infrastructure.

Organizations that do not understand the nuances of these trends are likely to fall behind.

Edited by Maurice Nagle