Data: What Industry Wants
February 18th, 2016
Editor’s note: Welcome to Throwback Thursdays! Every third Thursday of the month, we feature a classic post from the earlier days of our company, gently updated as appropriate. We still find them helpful, and we think you will, too! The original version of this post can be found here.
What should you do with data? That’s the overwhelming question facing industry today. Business has real problems that it needs to solve — integrating customer profiles across product lines, finding and mitigating manufacturing and supply-chain inefficiencies, understanding and addressing consumer churn, just to name a few — but often faces hurdles in putting data to work.
Since launching Silicon Valley Data Science, we hear three issues more often than anything else in the conversations we have with our clients: handling data overload, choosing analytic approaches, and having the right skills and resources.
Industry faces a broad spectrum of data challenges. The increased profile of data in business means that many want to know what big data really is, or if it’s necessary to hire a data scientist. Many of our customers understand the potential data has for their business, and have ambitions such as transforming legacy data architectures, or ensuring that new product development happens in a data-driven environment.
Across industry sectors, we’ve found that companies tend to have strategic data issues, ones that materially affect the business, requiring both long-term planning and near-term action. Today’s business leaders are well aware that they could be using their data better, and often have a strong idea of the benefits this would bring to the business.
There are three data challenges we hear more than others.
- We can’t capture the data we’ve got. While big data advocates tout the advantages of adding new data sources, many companies can’t yet exploit the data they’re already generating. They’re not yet at the stage where data can bring them a total picture of their business, and are having to throw away data from existing systems.
- What analytical approaches should we use? Capturing data is a big part of the battle, but the analytics use cases serve business needs and drive the underlying architecture. How can analytics be applied to existing systems, and how should new systems be architected to ensure we can fully take advantage of analytics?
- We don’t have the skills. Many companies don’t have the in-house capability to evaluate new data technologies and understand how they would address today’s business problems, or to plan for tomorrow’s needs.
So what should an aspiring data-driven business do to take steps forward? It’s a problem we’ve been thinking hard about as we assist industry in realizing their data ambitions.
We approach data-driven opportunities by first educating the business on the art of what’s possible — often a transformative stage as eyes are opened to the potential data can bring. We then move to a planning stage, identifying use cases that can demonstrate business or technical benefit. The best results always come from addressing a real need first. Finally, we use agile methods to quickly prove the viability of high-value use cases, and ultimately, move the new data capabilities through the pilot stage and into production.
Agile, cross-functional teams
The core of our approach is fielding teams that include business domain experts, data scientists, and engineers to create data-driven applications by iterating rapidly. In recent years, software companies have embraced the art of experimentation, using agile approaches to evolve requirements and product through iterative development. A similar approach works well for addressing data and analytic challenges.
Our agile approach opens up a number of ways to deal with the three main data challenges mentioned earlier.
When faced with overwhelming data, look for a part of the business that would benefit from having more data. Once a few candidate use cases have been identified, it’s much easier to envision which team, which data, and which technology can be used for rapid experimentation.
Quickly building out a well-defined vertical use case with new data—from data architecture through consumption to analytics—is an excellent way to answer questions about what analytical approaches to use. By building out a narrow use case that validates critical parameters for future planning, we’ve worked with customers to quickly assess not only a new data source, but also a candidate data architecture.
Lastly, bridging a gap in skills requires bringing in outside help. Especially at the early, exploratory stages, you want a team that brings an agile and collaborative approach to solution development, while also bringing both data science and engineering skills. Today, experimentation is cheaper and faster than ever, thanks to the flexibility of cloud-based software and platform services, such as Amazon’s AWS or Microsoft’s Azure. Pair that with the right partner, and you’re well on the way to driving your business further with data.
