Working Paper
Streaming Big Data in Finance
- Author: Skeggs, Richard.
The buzz word in business for the last couple of years has been Big Data. There is a wealth of presentations, papers and products within this area and it grows year on year. Most of these talk about ingestion, map reduce and data mining. The growth of streaming data produces its own problems and requires a new approach.
According to Forrester Research streaming big data analytics is defined as “Tools that allow a business to process and act on massive amounts of information while it’s still moving, as opposed to waiting for data to come to rest in a data warehouse or Hadoop. The technology is being used increasingly as new sources of data become common, such as streaming sensor data from the Internet of Things, streaming social media data like Twitter, and streaming mobile information from apps.”
The traditional approach to big data has been store the data then process. The processing can be performed in multiple ways and blended with other data sources to extract more meaning. This approach is not always possible with data streams. The questions then become:
- How and what can we extract meaning from the stream?
- Can we blend this with other datasets?
- Do we need to store the data for historic reasons?