Visualizing relationships between dynamic variables, part iii

Principal component analysis can be a powerful tool for detecting similarity between time series, and whether a series ceases to resemble one group of signals and begins to resemble another. I extract three principal components and create a three-dimensional dot plot. My function msPCABoxNF3[] used to label each dot with an Epilog, but with Mathematica 11.3 it became possible to use Callout[]s instead.

msPCABoxNF3[allChNF3sA, "names" -> "names", ImageSize -> Large]

We can show the evolution of these relationships over time using Manipulate[] (a snapshot of the tool is shown below)

msEvolvingPCABoxNF3[allChNF3sA, 20, "spacing" -> 200, "names" -> "tickers", ImageSize -> Large]

Mathematica can also export videos of these things, as shown below. The following runs a bit quicker than I would like but I’m not going to solve the problem for the sake of this post.

Visualizing relationships between dynamic variables, part ii

Mathematica’s Correlation[] tool is handy. I created a wrapper that adds labels and makes it a bit more convenient with the sort of data I use.

msCorrelationMatrix[allChNF3sA, "noisy" -> False]

I often like to consider correlation together with r^2, p-value and criticality tests. The following function runs these things simultaneously for convenience.

msCorrelationMatrix[allChNF3sA, "noisy" -> False]

These functions can also weigh recent events more highly than events in the distant past, which is typically how I use them, though this is not demonstrated above.

When sharing results with others, I find Weighted Adjacency Graphs to be useful, and I will often create tools that allow the user to scroll through time, watching given securities change what other instruments they most resemble. It can be particularly interesting to see ostensibly market-neutral hedge funds start to track the equity markets as their managers take on more beta. This often happens as assets under management increase and/or the markets have been performing well.

Here are three “snapshots” from such a tool for the securities above. You can see that at first, equity hedge funds are a sort of island that don’t track the performance of anything else in the graph. Then in 2005 we enter a regime in which equity hedge funds track the S&P, which in turn resembles other equity series, but credit and commodities follow their own path. The graph continues to evolve (I show only a few states below), but by the end we are in the current world of nearly all risky assets moving together, and nearly everything cross-correlating.

timePasses = Table[Map[msChangeChar[#, "data" -> EventSeries[Take[#["Path"], UpTo[i]]]] &, allChNF3sA], {i, 100, Length[allChNF3sA[[1]]["Path"]], 10}];
wags = Map[msWeightedAdjacencyGraph[#, .3, "time weights" -> Automatic, PlotLabel -> #[[1]]["LastDate"], ImageSize -> Large] &, timePasses];
Manipulate[wags[[i]], {i, 1, Length[wags], 1}]

Dec 2004

Jan 2008

May 2008

July 2012

It stays like that to the current day.

Tomorrow, principal component analysis.

Visualizing relationships between dynamic variables, part i

I frequently have to compare data series that evolve over time, looking in particular for how similar they are to each other, whether or not the behavior of a given variable changes over time, as well as the overall rate of increase, volatility-adjusted returns, and other metrics. A few thoughts below.

First, let’s load up the historical performance of an equity hedge fund index and a number of other financial time series (the S&P500, Barclay’s Global Credit Index, MSCI Daily Total Return Net World, etc.). Since I know I’m going to need it later, I’ll also compute the daily changes in these time series, and I’ll “align” them by selecting only the dates that all these series have in common.


totalReturnBenchmarkTickers = {"DJSMT Index", "XAU Curncy", "RU10GRTR Index", "LGDRTRUU Index", "SPXT Index",
"NDDUWI Index"}; hfNF3 =
msBBGhistory["HFRXEH Index", "Px_Last", DateObject[{2000, 1, 1}], "", "Day", "name" -> msBBGcurrent["HFRXEH Index", "Name"], "use DPDF" -> True];
bogiesNF3 = Map[msBBGhistory[#, "Px_Last", DateObject[{2000, 1, 1}], "", "Day", "name" -> msBBGcurrent[#, "Name"], "use DPDF" -> True] &, totalReturnBenchmarkTickers];
allNF3s = Prepend[bogiesNF3, hfNF3];
hfChNF3 = msChanges[hfNF3];
bogiesChNF3 = msChanges /@ bogiesNF3;
allChNF3s = Prepend[bogiesChNF3, hfChNF3];
allNF3sA = msIntersectionData[allNF3s];
allChNF3sA = msIntersectionData[allChNF3s];

As a first check, it makes sense to graph these against each other, normalizing so that they all start from the same place. I’m going to use Mathematica’s built-in DateListPlot[] function, including its built-in PlotLegends.
DateListPlot[msStartFrom1 /@ allNF3s, PlotRange -> All, PlotLegends -> Map[#["name"] &, allNF3s], PlotLabel -> "As a colorblind person, I hate graphs like this.\nEspecially if the legend goes to two columns."]

Edward Tufte recommends tagging each line with its own legend, which can be easily done using the Epilog option when graphing. It’s built-in to my function msEndTagPlotNF3[] (which also removes unnecessary extra framing and does other Tufte-like things.

msEndTagPlotNF3[msStartFrom1 /@ Take[allNF3s, 5], "rightpadding" -> 29, PlotLabel -> "I wrote this function years ago;\nit's better than Legend when the series don't end up near each other"]

msEndTagPlotNF3[msStartFrom1 /@ allNF3s, "rightpadding" -> 29, PlotLabel -> "But not great if the names start to land on each other"]

DateListPlot[Map[Callout[#, #["name"]] &, msStartFrom1 /@ allNF3s], Joined -> True, PlotRange -> All, Frame -> {True, True, False, False}, PlotLabel -> "Since Mathematica 11, I've often used Callout[] instead."]

Enough for today. Tomorrow I’ll show some tools for illustrating correlations and other descriptive statistics.

Edward Tufte weeps, part ii

Normally one has to turn to scientific literature for really terrible graphics, or USA Today. However, this morning’s Bloomberg Businessweek had a strong contender:

A graphic that makes it *harder* to understand what’s going on.

The x-axis is time, though kind of a pointless time, as what’s really being shown is the market’s reaction to six distinct Amazon product announcements. The Y-axis shows peak intraday drop in market cap (with the direction reversed — the more a stock fell, the higher its dot appears on the graph). The size of the dot shows the total loss of market value in dollar terms (so, not adjusted for the size of the company before the drop started). And the sizes are not a simple function of the total drop in value, but instead shows which of three categories a stock’s drop fell into. They’re labelled $100MM (or less, I presume), $500MM (everything between $100MM and $1B, I presume), or $1B (or more, I presume). Since the size of the dot is set at the end of the day, and location on the y axis is set intraday, these elements do not sync and are in some ways not comparable. The color of the dot indicates the industry in which each company operates and we are also given the name of each company.

What conclusion are we supposed to draw from this mess? The only trend I can see is that Amazon seems to be taking on bigger companies. Surely there would have been a more parsimonious, less absurd way of making that point.

Another unexpected dataset in Bloomberg

tickets =
msBBGhistory["BOXOSOLD Index", "Px_Last", {1980, 1, 1}, "", "Day"];
DateListPlot[tickets, Joined -> True, PlotRange -> All,
Frame -> {True, True, False, False},
PlotLabel -> "Box office tickets sold,\nU.S. and Canada",
FrameTicks -> {{Table[{i,
AccountingForm[Floor[i], DigitBlock -> 3]}, {i, .9*10^6,
1.55*10^6, 2*10^5}], None}, Automatic}]

Observe the internet happening