Exposing Dark Data for New Business Value

Anne Taylor

There’s considerable discrepancy around the volume of data that is unused, unstructured and hidden — aka “dark data.” Experts think organizations have a lot more of it than they recognize:

50 to 90 percent of data is dark, according to IDC, Gartner and others
25 percent of our data is dark, say IT leaders in a recent IDG survey

Despite their belief they have relatively little dark data, IT executives have big concerns about it, including the impact on quality, security and risk compliance, loss of control, lost business value, and the inhibition of data integration projects.

So, what is dark data? Gartner’s definition:

The information assets that organizations collect, process and store during regular business activities, but generally fail to use for other purposes.

Here’s a shortlist of dark data and the common places it lives:

Documents. Finance personnel often pull together expense or sales information from different sources, for instance, then do calculations on it inside spreadsheet docs. That number-crunching has no value if it’s sitting on someone’s desktop. The same goes for word documents, forms, PDFs and presentations. These files are often overlooked because it requires work to uncover what’s inside them, or there are misconceptions as to their value.

Data repositories. Lots of data — structured and unstructured — gets dumped into data warehouses, lakes and non-relational databases. These repositories often hold old records such as customer, employee or financial data that must be kept for compliance reasons, yet incur considerable storage costs. IT teams may struggle with the complexity involved to connect BI or analytics solutions to these repositories, and sometimes can only run analysis on individual silos — like the sales or customer relationship management app. Plus, they typically require vault-like access with few individuals holding the keys.

Applications. Consider how many emails contain attachments and where those files go. Then think about all the sensor data collected by Internet of Things applications; audio and image files sitting in social media apps; video surveillance; customer call records; log files. The Slack app probably contains great employee insights, but is any of that data captured, let alone analyzed, to improve user experiences or tap business opportunities?

Uncover dark data for business value

Pep Boys is a good example of what uncovering dark data can yield. The automotive retail and service chain wondered why some of its stores were underperforming. They integrated PBX system data into the Domo platform and analyzed, for example, how many rings it took before a service rep answered a call and the length of call-hold times. It quickly became apparent that better-performing stores had faster call pickup times and hence a greater probability of customers booking service appointments.

CIO Jarrod Phipps says his team never would have thought that PBX data had any value until they saw it in action. That one small project built data confidence and curiosity throughout his team.

“Now, we’re unshackled,” he says. “IT is less of a constraint, the business can operate more independently, and my team has probably 100 times the capabilities it had before. Nothing is off the table anymore.”

Next steps

No organization can solve their dark data problem all at once. Start by asking simple questions like: How do we arrive at our sales figures? What are the sources of all that sales data?

Then, get going with a small pilot project using a modern BI tool like Domo.

“The first day, we hooked up the connector and we had data up in production,” Phipps says. “The next day we took two years of historical data and validated it. And within two days, we not only had dashboards, but we actually had data assets that we had never contemplated being in a BI platform because it’s so hard to manage some of these old platforms.”