Big data—databases too large to manage by traditional means—was once a problem only for huge corporations, research universities and government agencies, but now even smaller companies grapple with it. Forward-thinking organizations will use smart methods to gain control of their big data and take advantage of the knowledge hiding inside.
As computing hardware and software technologies have advanced, gathering and storing astounding amounts of data has become easier and cheaper than ever. The tools to manage and analyze such large data sets and extract meaningful, actionable information from them have come along more slowly, and many organizations with big data on their hands lack the know-how to take advantage of it. However, by using well-known techniques, it is possible for all organizations to get a handle on their big data by laying the groundwork for exploiting the benefits.
The first step is taking stock of what you have, and this means developing metadata—information about the data. Compiling this information—such as the source of each table, or when it was created, by whom and for what purpose—will understand what the data could be useful for.
One important piece of metadata that must be established is the owner: the person or department responsible for each piece of data and for keeping it up to date and clean. “Clean” means not only that the data is correct and complete, but that each field has only data it have—phone numbers in phone number fields, names in name fields and so on. Like anything else, data without an owner becomes neglected, stale and eventually useless.
Once you know what you have, where your data is kept and who claims ownership, it’s possible to make the connections that lead to real information and knowledge. To take an overly simplified example, if you have daily sales volume data over a one-year period, and you have weather data over that same period, you can try to see if sales volume is correlated to the weather.
Putting Everything Under One Roof
To make these analyses, the data ideally must be housed together in what’s known as a data warehouse, which enables users to look at their data from any angle, perform analyses and draw conclusions. Constructing and maintaining a data warehouse requires some specialized expertise, so having staff or consultants who know they are keying.
Don’t Forget About Security
Another piece of metadata you need to know is how sensitive the data is. Are there credit card numbers, Social Security numbers, healthcare information, or other personal data lurking in there? If so, you must try to secure it under applicable standards. Don’t wait for a security breach to find out what you have to lose.
These activities cannot take place overnight; they require careful planning and execution. The information you get out of big data is only as good as the care you put into it.
Written by Dean Van Dyke, Vice President, Business Process Optimization
Dean Van Dyke is the Vice President of Business Process Optimization for iBridge. He brings more than 18 years of customer relations, business process outsourcing, lean six sigma, program/project management, records management, manufacturing, and vendor management experience to iBridge. Mr. Van Dyke was the former head of Microsoft’s corporate records and information management team and served honorably for over fourteen years in the U.S. Navy and Army National Guard. He received his Bachelor of Science in Business Administration from the University of South Dakota and his Master’s in Business Administration from Colorado Technical University.