Author Archives: stern

Visualizing relationships between dynamic variables, part i

I frequently have to compare data series that evolve over time, looking in particular for how similar they are to each other, whether or not the behavior of a given variable changes over time, as well as the overall rate of increase, volatility-adjusted returns, and other metrics. A few thoughts below.

First, let’s load up the historical performance of an equity hedge fund index and a number of other financial time series (the S&P500, Barclay’s Global Credit Index, MSCI Daily Total Return Net World, etc.). Since I know I’m going to need it later, I’ll also compute the daily changes in these time series, and I’ll “align” them by selecting only the dates that all these series have in common.

totalReturnBenchmarkTickers = {"DJSMT Index", "XAU Curncy", "RU10GRTR Index", "LGDRTRUU Index", "SPXT Index",
"NDDUWI Index"}; hfNF3 =
msBBGhistory["HFRXEH Index", "Px_Last", DateObject[{2000, 1, 1}], "", "Day", "name" -> msBBGcurrent["HFRXEH Index", "Name"], "use DPDF" -> True];
bogiesNF3 = Map[msBBGhistory[#, "Px_Last", DateObject[{2000, 1, 1}], "", "Day", "name" -> msBBGcurrent[#, "Name"], "use DPDF" -> True] &, totalReturnBenchmarkTickers];
allNF3s = Prepend[bogiesNF3, hfNF3];
hfChNF3 = msChanges[hfNF3];
bogiesChNF3 = msChanges /@ bogiesNF3;
allChNF3s = Prepend[bogiesChNF3, hfChNF3];
allNF3sA = msIntersectionData[allNF3s];
allChNF3sA = msIntersectionData[allChNF3s];

As a first check, it makes sense to graph these against each other, normalizing so that they all start from the same place. I’m going to use Mathematica’s built-in DateListPlot[] function, including its built-in PlotLegends.
DateListPlot[msStartFrom1 /@ allNF3s, PlotRange -> All, PlotLegends -> Map[#["name"] &, allNF3s], PlotLabel -> "As a colorblind person, I hate graphs like this.\nEspecially if the legend goes to two columns."]

Edward Tufte recommends tagging each line with its own legend, which can be easily done using the Epilog option when graphing. It’s built-in to my function msEndTagPlotNF3[] (which also removes unnecessary extra framing and does other Tufte-like things.

msEndTagPlotNF3[msStartFrom1 /@ Take[allNF3s, 5], "rightpadding" -> 29, PlotLabel -> "I wrote this function years ago;\nit's better than Legend when the series don't end up near each other"]

msEndTagPlotNF3[msStartFrom1 /@ allNF3s, "rightpadding" -> 29, PlotLabel -> "But not great if the names start to land on each other"]

DateListPlot[Map[Callout[#, #["name"]] &, msStartFrom1 /@ allNF3s], Joined -> True, PlotRange -> All, Frame -> {True, True, False, False}, PlotLabel -> "Since Mathematica 11, I've often used Callout[] instead."]

Enough for today. Tomorrow I’ll show some tools for illustrating correlations and other descriptive statistics.

Edward Tufte weeps, part ii

Normally one has to turn to scientific literature for really terrible graphics, or USA Today. However, this morning’s Bloomberg Businessweek had a strong contender:

A graphic that makes it *harder* to understand what’s going on.

The x-axis is time, though kind of a pointless time, as what’s really being shown is the market’s reaction to six distinct Amazon product announcements. The Y-axis shows peak intraday drop in market cap (with the direction reversed — the more a stock fell, the higher its dot appears on the graph). The size of the dot shows the total loss of market value in dollar terms (so, not adjusted for the size of the company before the drop started). And the sizes are not a simple function of the total drop in value, but instead shows which of three categories a stock’s drop fell into. They’re labelled $100MM (or less, I presume), $500MM (everything between $100MM and $1B, I presume), or $1B (or more, I presume). Since the size of the dot is set at the end of the day, and location on the y axis is set intraday, these elements do not sync and are in some ways not comparable. The color of the dot indicates the industry in which each company operates and we are also given the name of each company.

What conclusion are we supposed to draw from this mess? The only trend I can see is that Amazon seems to be taking on bigger companies. Surely there would have been a more parsimonious, less absurd way of making that point.

Another unexpected dataset in Bloomberg

tickets =
msBBGhistory["BOXOSOLD Index", "Px_Last", {1980, 1, 1}, "", "Day"];
DateListPlot[tickets, Joined -> True, PlotRange -> All,
Frame -> {True, True, False, False},
PlotLabel -> "Box office tickets sold,\nU.S. and Canada",
FrameTicks -> {{Table[{i,
AccountingForm[Floor[i], DigitBlock -> 3]}, {i, .9*10^6,
1.55*10^6, 2*10^5}], None}, Automatic}]

Observe the internet happening

Printing to a 1980-era daisywheel printer from OS X, part 4


If everything is working, you should now be able to print. Feed a piece of paper into the printer and send a test print from the Terminal with, for example echo "this is a test" | lpr -P cables2go_IEEE-1284_Controller

Those unfamiliar with printers of this era — you may need to take the printer off-line to advance the paper, then put it back on-line to print. The off-line/on-line toggle goes by different names on different printers; on the NEC it’s just called “print”.

For text files of any length, simply piping the files to the printer will probably not produce the output you want — these printers do not recognize when they have reached the end of the page and will cheerfully print past the end of a line. Further, if told to go to a new line, but not to back up to the left, they will produce bizarre laddered text, and if told to go back to the start of the line but not to advance the roller, they’ll print right on top of the text they just printed. There are two solutions to this:
1. force line-ends in the software that creates your files. Most sophisticated text editors can do this. In a normal typeface, 80 characters will fit neatly across an 8.5” page. It is important that you save such files in ‘DOS’ format, which includes both carriage return and linefeed commands at the end of each line. If you can not save in this format, don’t bother forcing line-ends at all; they won’t work. On the Mac, I use TextWrangler.

2. Alternatively, you can pipe the file through a UNIX utility that breaks your text into lines (fold -s) , and then through a second command that converts the breaks into the carriage returns and linefeeds the printer needs. There are dedicated utilities that you can install to do this latter task (dos2unix), or you can use awk another similar, more general translation tool.

I generally use the latter path, printing the file example.txt with a command like

cat example.txt | fold -s | awk 'sub("$", "\r")' | lpr -P PrintAdapter

Forcing line breaks as in the first proposed solution can create a bit of a hassle with later edits; the fold -s | awk | lpr solution is therefore more general and typically easier.

The following idea gives some idea of the interaction of file format and output processing

line-end forced linebreaks fold-s | awk | print
CR bad bad
LF bad good
CRLF good good