White Papers Making more of data using AI | Page 5

Data Management

A ComTech Advisory Whitepaper

Figures 2 and 3: Number of Data Sources Used and Increase Over the Last Two Years
0%

1 to 10

14%
32%

9%

11 to 25
25 to 50

9%

51 to 100
36%

100+

Less

4%
23%

18%

Same

55%

Less than
doubled
Doubled

The quality and accuracy of data varies significantly depending upon its source and much work
often needs to be performed on the raw data to validate, check, potentially convert, derive
calculated values and much more, before it may be deemed to be useful. Multiple versions of data
sets also need to be stored and maintained across date ranges and to include any and all changes
made to the data to ensure repeatability. The types of data that traders work with are also
increasingly diverse. The survey showed what types of data traders used and the relative value of
that type of data to the commodity trader (Figure 1).
While tools and services exist to help users with data and data management, the survey also
showed that internal IT still plays an important role in data management. The majority of
respondents said that their internal IT normalizes or structures data for the traders. However, a
common complaint across the respondents was the amount of time spent cleaning and preparing
data as opposed to actually analyzing it. They also complained of the lack of good data
management techniques and tools available to them generally. As the quantity of data increases,
these issues will surely only get worse.

Social Media
Social media includes tools like Facebook and Twitter generating vast amounts of data every
minute. It also includes the blogosphere where experts and amateurs alike post their views. Only
a small amount of this ‘data’ might be of use for traders however. An example of such use could
include local weather and emergency conditions tweeted by residents such as perhaps,
transportation disruption in a producing area that may have an impact on price formation due to
reduced supply availability. Social media could equally well include broker’s blogs or the writings
of trading commentators.
The issues with social media though include its reliability (much of it is opinion based on little of
no factual analysis) and its sheer volume. Despite that, there are a number of companies already
offering social media derived data products and a number of generic news platforms are now
incorporating it.

© Commodity Technology Advisory LLC, 2014
5