Big data—databases too large to manage by traditional means—was once a problem only for huge corporations, research universities and government agencies, but now even smaller companies grapple with it. Forward-thinking organizations will use smart methods to gain control of their big data and take advantage of the knowledge hiding inside.
As computing hardware and software technologies have advanced, gathering and storing astounding amounts of data has become easier and cheaper than ever. The tools to manage and analyze such large data sets and extract meaningful, actionable information from them have come along more slowly, and many organizations with big data on their hands lack the know-how to take advantage of it. However, by using well-known techniques, it is possible for all organizations to get a handle on their big data by laying the groundwork for exploiting the benefits.
The first step is taking stock of what you have, and this means developing metadata—information about the data. Compiling this information—such as the source of each table, or when it was created, by whom and for what purpose—will understand what the data could be useful for.
One important piece of metadata that must be established is the owner: the person or department responsible for each piece of data and for keeping it up to date and clean. “Clean” means not only that the data is correct and complete, but that each field has only data it have—phone numbers in phone number fields, names in name fields and so on. Like anything else, data without an owner becomes neglected, stale and eventually useless.
Once you know what you have, where your data is kept and who claims ownership, it’s possible to make the connections that lead to real information and knowledge. To take an overly simplified example, if you have daily sales volume data over a one-year period, and you have weather data over that same period, you can try to see if sales volume is correlated to the weather.
Putting Everything Under One Roof
To make these analyses, the data ideally must be housed together in what’s known as a data warehouse, which enables users to look at their data from any angle, perform analyses and draw conclusions. Constructing and maintaining a data warehouse requires some specialized expertise, so having staff or consultants who know they are keying.
Don’t Forget About Security
Another piece of metadata you need to know is how sensitive the data is. Are there credit card numbers, Social Security numbers, healthcare information, or other personal data lurking in there? If so, you must try to secure it under applicable standards. Don’t wait for a security breach to find out what you have to lose.
These activities cannot take place overnight; they require careful planning and execution. The information you get out of big data is only as good as the care you put into it.
Written by Sofia Johnson, Manager, Software Development
Ms. Johnson, an expert in Project and Resource Management, is the Software Development Manager at iBridge. She brings 11 plus years of IT work experience and business intelligence to provide successful customer engagement of software development. Prior to working at iBridge, Johnson worked as a Senior Engineer for Hewlett Packard and Oracle, and a Hyperion Consultant for IBM and Google.
She is a product expert in enterprise contract management software solutions – diCarta/IBM Emptoris. As part of her previous engagements as a Hyperion Consultant, she made significant contributions to optimize and enhance a BI/Analytic solution for a major food, health and home retailer, LoBlaw, in Canada. She introduced performance tuning and optimization principles to the existing solution by leveraging Essbase cube partitioning techniques and re-writing some of the calculation logic to bring in significant performance improvements. Another significant engagement included automation and enhancement deliveries of a Hyperion analytical solution for a South African multinational brewing and beverage company (SABMiller) headquartered in London, England.
Ms. Johnson is a certified Essbase developer with a Master’s degree Computer Applications from Bangalore University. She has immense passion for travel, reading and working for social causes.