Recently I was pleased to be invited as a keynote speaker to the SGPE conference at Crieff Hydro, Crieff, Perthshire. The SGPE is a joint collaboration between eight Scottish Universities (Aberdeen, Dundee, Edinburgh, Glasgow, Heriot-Watt, St Andrews, Stirling and Strathclyde) to provide Masters training to students. The conference is centred on training for both MSc and PhD students and it is an exciting time to bring them altogether.

During my keynote session I discussed the opportunities that Big Data can offer applied economists. After briefly defining Big Data and providing a primer on basic analytics techniques (like text analytics, social media analytics and machine learning) I drew on examples drawn from economics, business and the work of the BLGDRC research team. The main gist of the talk was that Big Data can offer many opportunities as they will allow you to take into account qualitative phenomena into quantitative analysis, work on more granular data with the results and make observable the “unobservable”.

However, these benefits come with challenges that need to be taken into account. These data have often not been collected for research but are instead just operational data. Therefore the quality and the suitability of data need to be assessed before they can be used for research. Inference on these data is not well-understood and issues around the representatives of the sample (or whether we have a sample or the population) need to be articulated better by the scientific community. Scalability of techniques from small samples to large samples needs to be assessed as well as not all techniques that have been designed for small samples can be adapted to large samples. In addition, some of these data do not have documentation which makes the tackling of these issues more difficult.

The use of Big Data is very different to the way an economist would usually carry out research. The standard practice for an economist would be to start with a vision, test this vision and then decide if it is true or false. However in data science it occurs the other way around. Instead we analyse data to find patterns which may then inform the development of a theory. During the conference there was a debate about the extent to which economists could and should make use of this new way of working, as they do not want to lose the benefits of their existing methods. In my view economists should still retain the theoretical view as a first port of call, and generally the other professors in attendance agreed. However there is of course a lot to be said for the analysis of Big Data and therefore we must also allow for these potential benefits to be realised.

I am sure this debate will continue and hope that for those who were part of the SGPE conference, and for all those who read this blog post, I get them thinking about the benefits of the new techniques. At the Business and Local Government Data Research Centre we are pioneering the use of these techniques alongside the rest of the Big Data Network. Please get in touch with us if you’d like to find out how Big Data could help you, your organisation or your business.

Blog post by Professor Vania Sena (Director of the Business and Local Government Data Research Centre), please if you have any questions about the contents of this post.

Published 26 January 2017