We Need a New Data Architecture: What Next?
October 20th, 2016
Editor’s note: Welcome to Throwback Thursdays! Every third Thursday of the month, we feature a classic post from the earlier days of our company, gently updated as appropriate. We still find them helpful, and we think you will, too! The original version of this post can be found here.
You’ve realized that your business needs a new data architecture, but what next? Data systems serve more stakeholders than ever before; new technologies constantly become available, and competitors are moving faster than ever. How do you make the right decisions and manage the risks of moving to a new data platform? The key is in understanding the attributes of a good modern data architecture, and in adopting a data-centered view of your business.
The challenge of moving forward
Enterprise IT faces a pressing need to understand the new data architectures required to support business. The demand is universal, due to massive volumes of data from internet and mobile applications; the need to generate competitive advantage from new types of information, such as streams from Internet of Things sensors, images, or social media; and the expectation that we create the kinds of friendly online analytical products that users are now familiar with thanks to Google, Facebook, and other web services.
It’s clear from the explosion of interest in newer platforms and technologies that the old tools and licensing costs don’t work to meet new business needs. Open source, cloud, and scale-out distributed systems create a new cost model. IT needs to know how and when to use platforms such as Hadoop, Spark, AWS, Azure, or Google Cloud.
The path forward isn’t so clear, however. It’s reasonable to be worried about getting your fingers burnt. For example, some early adopters rode the NoSQL wave with enthusiasm, but discovered that no single type of database met their needs: many such databases come with hefty requirements for extra programming. They have since reverted to relational databases or, more commonly, adopted a hybrid architecture.
The world has moved into a business technology space-race era. It’s not enough to support stable business processes; IT must also support innovation and iteration. Early adopters in that race reap the rewards of trying audacious new things—but they also bear the non-trivial costs when some of those things explode.
As a modern business, your challenge is to move data infrastructure towards creating a platform that sustains today’s business needs, the innovation process, and future use cases—all while managing the risks of the unknown, and delivering valuable results to the business.
Managing the journey towards new data technologies
At SVDS, we hone our approach through R&D, then export the learnings of early adopters in a way that makes sense at enterprise scale. Adopting and engineering new data platforms is an inescapable requirement for most businesses, and we have devised a method for creating modern data architectures that work, even in the face of rapid change.
A good modern data architecture:
- supports current and future technical capabilities,
- considers fit with existing architecture,
- selects the appropriate technology platforms, and
- delivers a plan for implementation.
Being adaptable and future-proof means that you need to spend a lot of time considering what we call the “data value chain”—the stages of data as it enables your business: discovery, ingest, processing, persistence, integration, analysis, and exposure. Your current and future requirements at these stages often have the largest influence on technology selection.
For example, the need for real-time analytics not only mandates certain performance requirements for data processing, but also requires service guarantees from data ingest, and has an effect on how the results of those analytics are exposed back to the organization.
By thinking data-first, rather than application-first, you can avoid the costly data silos that prevent so many businesses from leveraging their data. And when new technologies emerge, as is inevitable in today’s fast-evolving market, you have a strong vantage point from which to judge them.
Business comes first
By far the biggest factor in any architecture analysis is the needs of the business itself. An effective modern CIO creates an enabling platform for the business to innovate and build upon. The IT mindset must transition away from policies governing users, towards creating tools that enable them.
That’s why you just can’t “get a Hadoop” and make everybody happy: simply installing a tool and kicking the tires doesn’t translate into a serious-minded exploration of your present and future data needs. In other words: modern data architectures are not a shrink-wrap problem. If it were that easy, then the business advantages wouldn’t be so radical.
Though challenging to get right, the ultimate benefit of a modern data platform is to foster innovation and make business more agile. If you don’t transform into a data-driven company, you become deeply vulnerable to data-driven competitors. To dive deeper into the kind of business advantage a modern data architecture makes possible, read my essay on building an Experimental Enterprise.