Deriving Business Value from Big Data using Sentiment analysis A ComTechAdvisory Whitepaper
BIG DATA AND ITS POTENTIAL APPLICATIONS
A couple of years ago, ComTech undertook some research in the commodity trading and risk management arena around the value of social media as data. The results were not particularly startling in terms of the interest levels expressed in something so chaotic and voluminous as social media 1. Yet, when it was explained in the context of a meaningful example, interest levels heightened considerably. Since then, many big data opportunities and application areas within commodity trading and risk management have been identified and they encapsulate a whole raft of new possibilities including, for example
/ Trading Analysis
- Predictive analytics
- Pre-trade decision support analytics including sentiment analysis
/ Market Risk
- On demand risk management
- Predictive indicators
- Exposure simulations
/ Regulation & Compliance
- Trade surveillance
- Fraud management
- Regulation / Compliance audits
Many of these areas are already being actively explored and productized. For example, Bloomberg announced that it is enhancing its enterprise compliance platform to provide next-generation communications surveillance functionality and analytics to meet increasingly stringent regulatory guidelines, prevent market abuse, and deepen visibility into the commercial use of social media. The financial trading dashboard managed by Thomson Reuters is another example, and it uses sentiment analysis data to track news on 20,000 stocks and thousands of commodities. It parses text from multiple sources, looks for keywords, tone, relevance and freshness to provide sentiment analysis for traders to act upon.
Indeed, sentiment analysis has developed rapidly as a technology that applies machine learning and makes a rapid assessment of the sentiments expressed in the various types of unstructured data available today in the form of social media, news and blogs. These sources of information can move the market and are measured quantitatively. Analysts and investors digest financial news and their perceptions can rapidly impact the market and move stock and / or commodity prices.
However, making use of masses of unstructured data of variable quality and reliability isn’ t an easy undertaking. It requires a high degree of specialism, usually provided by data science, and those with the expertise to deploy the right combination of analytics, machine learning, data mining and statistical skills as well as experience with algorithms and coding in order to explain the significance of data in a way that can easily be understood by others. Part of the problem is in understanding exactly what is meant by the words used by social commentators and others. This means dealing with synonyms, spelling errors, use of different languages such as Latin, polysemy( where one word can actually have many meanings) and the sheer volume of data, amongst other issues. If these issues can be resolved, then one is left with the ability to track brand perception and business opinion trends that might have real business value.
The types of data that can be analysed using this approach include news feeds from almost any source, as well as social media content from popular tools such as Twitter. Taking the data in near real-time, stripping out noise and irrelevant content and using Natural Language Processing( NLP) and machine learning in an attempt to extract useful meaning can result in information that has immediate and actionable value in the form of, for example, sentiment analysis.
1) Making More of Data Using AI, Commodity Technology Advisory White Paper, 2014
© Commodity Technology Advisory LLC, 2015, All Rights Reserved.