Building a Software Defined Data Center: Automation, Orchestration and Agility

Despite the emergence of IT as a Service and the rise of self-service catalogues, most IT operations—including EMC’s—have remained largely manual when it comes to filling users’ requests for networking, storage and compute, struggling to keep pace with growing demand. Until now, that is.

EMC IT is in the process of rolling out a new set of tools, based on a combined approach to infrastructure and automation that will reduce the time it takes to fill customers’ infrastructure demands from months to days or even hours.

The new production environment uses EMC’s Federation Enterprise Hybrid Cloud (FEHC) management platform on VCE Vblock™ converged and hyper-converged infrastructure to provide the abstraction of hardware through software. Translation: IT clients will no longer have to come to the IT infrastructure team every time they need a new environment or an additional server. They can self-provision these services using a truly automated portal and with a standardized set of components.

EMC IT is initially launching the new FEHC model to a limited number of internal users in IT, starting with providing non-mission-critical cloud services to Cloud Infrastructure Services.  We will then progress to providing mission-critical, middleware and database services. The roll out is a first step towards our planned launch of a software defined data center that is slated to automate and orchestrate 50 percent of our 30-petabyte data center workload by year’s end.

With this new approach, our internal IT customers will be able to provision a new virtual machine (VM) in an hour rather than the three weeks it took previously using the traditional, manual IT process—certainly something that most organizations’ IT operations are striving to achieve.

But how did we get here and why should it matter to your organization?

Industry-wide, the need for IT automation and orchestration is recognized as key to achieving the speed and agility that IT users require. Among the goals of automation:

1. Reducing deployment time. How long customers wait for service impacts your organization’s return on investment (ROI) and service level agreements (SLAs). Reducing manual labor is crucial.

2. Reducing complexity. Traditional manual IT processes can lead to the proliferation of home-grown processes with a variety of machines stemming from doing things differently each time. With automation, we strive for standardized processes using vendor-based application programming interfaces (APIs) to provide for machine-to-machine communications.

3. Creating standard and repeatable processes. This is a hard one. You have to go through the task of identifying your standard processes so you can repeat them. People are quite used to dealing with manual processes such as spread sheets and emails. When I come in and create block diagram of the processes and identify APIs that will reduce manual touch points, it is a big adjustment for them.

4. Creating a self-service process. Once you have achieved the previous three goals, you gain the ability to scale quickly, create use cases, onboard more customers faster and create an overall better customer experience. Now they can go to a portal and request a catalog item with just a few clicks.

5. Getting people to adopt the new model. We can build automated services but if no one uses them, they are worthless. That is the challenge EMC IT is currently facing—capturing the minds of our users to change the culture toward automation. Users and IT experts need to trust that the automation process will build out what they need instead of relying on a person as they always have.

These goals cover why automation is needed, but how would your organization get started to build the green field environment to then make the transformation to a software defined data center?

In our case, this involved transcending the traditional siloes of IT services and collaborating on defining user needs. For the past six months, EMC IT subject matter experts in automation, network, storage and compute have been working together to create software and services that will automate the way we manage IT infrastructure.

We began with a week-long planning session to identify our use cases and create a log of what services we were going to need in order to roll out this new model to our internal IT customers. We needed architects from the four groups to be on the same page regarding what the transformation to automation and orchestration would look like.

Next, we created an agile work stream, using a DevOps approach in which software people (like me) worked directly with our counterparts in compute, network, and storage to develop software and operations processes. Teaming up with data center operations, we started with a smaller, pilot-stage of DevOps where we created services, reviewed them with our internal customer to get feedback, and then made changes.

It was an evolutionary, agile process for the development — prioritizing services and developing them in a series of sprints. At the same time, the network, storage and computer team were working on their own architecture. We worked in parallel and bridged their processes where needed. Our goal was to publish fully compliant Windows 2012 Core, Windows 2012 GUI and Red Hat Linux VMs, as well as VIPR Software Defined Storage which we’ll be offering to our initial customer group in Q3.

We are building out a new environment in our Software Defined Data Center for EMC IT where all the automated requests for VMs, storage and networking will be handled. We plan to use this distributed model to integrate all the different software APIs then orchestrate the code library, removing the need for manual processes.

The result is similar to building an airplane on a factory line compared to building one by hand: By hand, even a small plane would take a year to build and be difficult to test pre-flight, but if you build it using a mile-long hanger and automated assembly, you can create a much higher quality airplane in 50 days. And you’ve done it using a repeatable process.

It is a longstanding concept that makes sense for today’s fast-moving IT infrastructure needs.