30 January 2012

Big data for development

Communications can provide insights into poverty, so studying communications patterns can be helpful for designing development policies and programmes. That’s just one point emerging from a study at Harvard University’s Engineering Social Systems programme which is looking into ‘big data’, the new buzz word in academia, government, development and media circles.

And they are not alone. Academics from New York University and the University of New South Wales have published a series of ‘provocations’ to spark discussion about access to data and how it is used. “Will large-scale analysis of DNA help cure diseases? Or will it usher in a new wave of medical inequality?”

The United Nations has launched the project Global Pulse, which aims to evaluate masses of data in order to generate early warning signals about developmental challenges, ranging from crop failures to social discontent that can trigger civil conflict.

Big business is also hot on the trail. Mckinsey & Co. have published a paper on ‘the next frontier of data’. Of note, they don’t define big data by the size, since what is big today will be manageable before long: “big data refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze”. With the benefits of exploiting such data predicted to exceed $300 billion to the US health sector annually, it is no wonder McKinsey is taking interest.

Ultimately it is all about you and me. Big data reflects the increasing capacity to track our every move, especially online. From spending patterns and vacation histories, to web browsing and more, the interests and habits of the greater public have never before been so well documented. It is opening a big opportunity for planners and policymakers, as well as marketers, strategists and also those who would exploit the information for less agreeable purposes – legal or otherwise. It is thus important and potentially very valuable that legislators and regulators both support open access to data for legitimate research, but also necessary to protect personal privacy, safeguard the integrity of data and set standards on its collection and the application of findings.

Further reading of interest: Bloomberg Business Week, on the dangers of insufficient information.
Publisher Tim O’Reilly on Google+ reports on a recent discussion of success factors, including the power of visualising  data.