It’s called “dark” data for a reason: It’s hidden in places like individuals’ desktops, unused databases or other repositories. It’s buried in documents like spreadsheets, and inside unstructured sources like audio or video files. And dark data can also refer to data that sits outside a governed process, which limits business users from doing their job.
Unfortunately, if you can’t access it or determine its value, dark data can present both risks and rewards, according to members of IDG’s Influencer community of IT professionals, industry analysts and technology experts.
“Dark data for some organizations is akin to sitting on an untapped oil well,” says Jason James (@itlinchpin), CIO of Net Health. “It is there taking up space and, depending on the type of data, could pose a risk. Much like an untapped oil, that data could be extremely valuable once refined.”
The challenges with dark data
Data overload and the lack of the modern solutions to tackle it are among the challenges organizations face.
“While all organizations look for opportunities to closely analyze this data at a later date, many do not get around to it,” says Scott Schober (@ScottBVS), President and CEO of Berkeley Varitronics Systems.
Unfortunately, when that data just sits there, the risks grow.
“Dark data is an asset until it becomes a liability,” says Nick Gonzalez (@nickg1421), Senior Technology Marketing Manager. The right analytics can turn dark data into insights for monetary gain, “however, the more stored, the higher risk you carry.”
Also, there is little incentive to process and analyze the data now.
“As storage costs have decreased dramatically over time, businesses have increased the level of data that they collect and store over the years, leading to huge pools of dark data within enterprises,” says Larry Larmeu (@LarryLarmeu), Financial Services Lead at Accenture.
Yet letting this information remain stored, idle and hidden can lead to “severe consequences,” he says. Privacy initiatives such as the General Data Protection Regulation and the California Consumer Privacy Act should cause organizations to reconsider keeping data in the dark.
For example, among the consequences of an organization’s insufficient privacy measures is that consumers will take their business elsewhere.
“Dark data is collected during regular business activities and according to Gartner, it is generally is not used. Sounds harmless but consumers do not believe it is harmless,” says Debra Ruh (@debraruh), CEO of Ruh Global IMPACT.
The rise in data breaches, leaks and insufficient security measures has caused individuals to lose trust with organizations, Ruh says. “The public’s perception of data including dark data being inappropriately used will continue to grow.”
And then there’s the security challenge.
“The big issue I see with untapped dark data is that it easily becomes a vulnerability,” says Will Kelly (@willkelly), Technical Marketing Manager. “Another issue is a hacker coming across dark data they find interesting and the theft opens a new attack vector on an organization.”
The hidden value in dark data
Exposing dark data not only helps avoid business risks, it can also shed a light on potential business value.
“Much like the rendered fat from Sunday bacon cooked slowly in the oven, many companies have no idea of the value of the unstructured goodness that is dark data,” says Gene DeLibero (@GeneDeLibero), Chief Strategy Officer of Geekhive. “It can provide innovative businesses with the context necessary to unlock trends, patterns and relationships they might otherwise miss.”
Organizations must tap into this dark data to figure out its worth, says Steve Prentice (@StevenPrentice), Technology Integration Specialist.
“The issue at the moment is a value question — weighing the cost of storage against its potential future value,” he says. “It’s the digital equivalent of having a warehouse full of residual ‘stuff’ that might have the potential for being sold sometime in the future, but for now is just collecting dust and costing money.”
Prentice advocates for not putting a time limit on data, rather turning to emerging technologies to plumb for value.
“Ongoing developments in artificial intelligence, machine learning and big data analytics, paired with ever-expanding computing power means that patterns and information hidden inside dark data is more like gold hidden inside a mountain, rather than useless dirt,” he says.
Next steps
Organizations should take steps now to shed a light on dark data — exposing both its risks and rewards.
“Given the massive amounts of unstructured data, a major issue is first analyzing an inventory of ‘what dark data we already have,’ as opposed to ‘what insights we’d like to get from it,’ regardless of the dark inventory,” says Frank Cutitta (@fcutitta), CEO and Founder of HealthTech Decisions Lab.
Larmeu agrees: “Within dark data may lie many secrets about our business and our customers, but in order to take advantage of this we need to improve practices around identifying and classifying data sources, enabling us to unlock the power held within.”
Just getting that far can be a daunting task for most organizations.
“In order to maximize dark data’s potential and minimize the inherent liability of storing so much valuable data, companies have turned to third parties with vast resources and a fresh perspective on analysis to open up new monetization possibilities,” says Schober.
The key is to get started so that dark data can be used to “drive better business decisions or even create new lines of business based on the type of data,” James says.
A modern approach to exposing dark data
Organizations should start with a unified modern data platform to automate the ingestion, cleaning and blending of data. For example, Domo leverages tools and technologies to bring dark data to light, such as through ETL data pipelines, connectors and orchestration. In addition, it provides self-service functionality so users can quickly access data in the context that is most relevant for them.
Start solving the dark data problem. Visit here.