Category Archives: mathematica

Quine in Mathematica

A “quine” is a curious object in computer science of far-reaching implications. It is a program that generates itself as output. This is more complex than it sounds, as if you have your program source, to print that source requires that the program have something like Print[program source], but then to capture that leading print command, you need to amend your code to something like Print[“Print[”,program source,“]”], and to capture the new complexity you need another level of nesting, ad infinitum. Some people, discovering this problem upon their initial efforts to write a quine, conclude the problem is hopeless.

However, it is not at all hopeless. These can be created in every Turing-complete computer language, and in fact can be created an infinite number of ways in each language. I learned about quines recently from their Wikipedia page, and from a very nice discussion at the web page of David Madore. Reading Madore’s page, I immediately realized an extremely slick implementation that would be possible in Mathematica 8, but before trying to implement that I thought I would try a more general approach first.

Three different techniques seemed promising. The first involved putting a program that uses ToExpression[] to render a string into a string, and then run ToExpression on it. My efforts in this direction all hit infinite recursion loops.

The second direction involved using StringReplace to print a string that contained its own StringReplace command. The substring being replaced seemed to get Held, however, and I couldn’t get it to behave as desire.

The third way uses the Print command, thusly:

quine[x_String] := Print[x, ";", FromCharacterCode[10] , "quine[", InputForm[x], "]"];

The output is precisely as we want —

I think this meets the definition of Quine. I’ll set to the elegant Mathematica 8 only version shortly.

Another old computer — the Powerbook 180c

In 1993, Apple released the PowerBook 180c, which sold for over $4100 new. It was, I think, the final PowerBook with the iconic form factor that had been introduced with the PowerBook 100 in 1991.


Somewhere along the way I picked one up for a percentage point or so of the original selling price. The memory had been upgraded to 14Mb, an amount that would have been astounding at the time, and almost unimaginably expensive, considering the Sumitomo Chemical Plant fire of July 1993 that caused RAM prices to spike. Consider this in comparison to the recent Tōhoku earthquake and tsunami, which put much of northern Japan at least briefly underwater and had virtually no impact on the global economy. In 1993, a fire in one Japanese factory caused computer memory prices to remain elevated for two years.

about this mac 1993

I hauled the computer out of the basement recently to convert some old floppy discs, and put it through some paces online. Unlike the Apple Lisa I recently connected, indirectly, to the web, this machine has a true TCP/IP stack, and can run mail clients, web browsers, and the like.

The machine was made before Apple routinely put ethernet ports on their computers, so an adapter is needed. There probably exist AppleTalk-to-Ethernet adapters that can be connected to the machine’s serial port. I am lucky enough to have an Asante Mini EN/SC 10T adapter, which allows older Macs to be connected to ethernet via their SCSI ports. The driver software was well coded, and the Mac regards the resulting connection as built-in ethernet.

asante en/sc 10T

Network access in Mac OS7 was handled via the MacTCP control panel, not the network control panel, which back then related only to the AppleTalk protocol I think. If you have a similar device and are having trouble configuring it, select “Ethernet Built-In” in MacTCP, then click the “More” button. If you plan to talk to the world, find the names and addresses of your DNS servers and enter them by hand. Your ISP can tell you, if you don’t use OpenDNS or another alternative.

Set your router address (called a “gateway address” here).

I do not think MacTCP handled DHCP correctly. Anyway, I set a manual IP address just in case.

macTCP 1993

Numerous older web browsers and FTP clients, etc., are available online. I used NCSA Mosaic and Fetch. Then I logged into IRC using the venerable Ircle client, which unfortunately killed itself after 30 minutes to punish me for not buying a license.
irc on a PowerBook 180c

Mathematica 2.2 was loaded on the machine, and I ran a couple of quick problems through it.
mathematica 2.2, 1993

The syntax for simple problems like this has not changed, and those problems can be solved using the exact same commands in Mathematica 8.01. On my home Mac Pro, the first problem can be solved a bit better than 8x as quickly on the modern machine, and the second problem about 127x as quickly.

Using NCSA Mosaic, here’s how this website would have looked in 1993:

monkeywrench as rendered in NCSA Mosaic

A silly article from Reuters

Reuters ran an article on the 19th titled “Wall Street’s ‘Buy Everything’ Sentiment Continues” which took as its premise that stock prices in the U.S. are rocketing upwards at an unprecedented pace, and that this, combined with low trading volumes and high levels on the VIX index, means that stocks are “due for a correction”. They quote only one person in the article, Paul Mendelsohn, sage of Charlotte, Vermont. He makes the specific claim that he has “never seen a market like this,” though he has been a “market watcher” for 35 years.

There may be reasonable explanations. Perhaps stocks were too low, so even rising rapidly, they may still be cheap. Or perhaps corporate profitability has increased significantly, so that stocks must rise rapidly to remain fairly valued. Mendelsohn does not consider such issues, saying “I’m showing, by every technical and quantitative standard I have, this market is at extreme levels. But no matter where we start out in the morning, buyers come in.”

Sadly, he doesn’t tell us what “technical and quantitative” standards he has, but the article does give us some clues of what might be worrying Mr. Mendelsohn. It claims that “Trading volume has been exceptionally low recently and the CBOE Volatility Index .VIX, Wall Street’s so-called fear gauge, is up on the week despite the gains in stocks. The index is usually inversely correlated to the S&P 500, and a rise in the VIX typically means a drop in the stock market.”

Hmm.. that sounds wrong.

normal volume

That goes back as far as Bloomberg has the data, and recent volumes don’t look unusually low to me at all. It looks pretty normal, and that volume data is from the New York Stock Exchange, which has been losing share to competing organizations like Liquidnet for years, so total recent volume is certainly higher.

How about the supposedly torrid pace of stock price increases? Reuters worries that “Wall Street posted its third consecutive week of gains with the S&P 500 now up 6.8 percent for the year and more than 20 percent in just six months.”

How weird is this? Let’s consider the 35 years of Mr. Mendelsohn’s expertise.

pretty normal
pretty normal too

It feels hard to panic over that; the current situation looks pretty much like any other bull market. Volume is normal, the pace of increase has plenty of precedent. Through most of the previous periods that looked like this, you’d have been very happy to continue to hold the market.

Oh, about the last claim, that an increase in VIX predicts a drop in the stock market? I have heard this from others, and rigorously testing it is complex (what is the proper lag between an increase in VIX and a drop in the market? Do we care about the absolute level of the VIX, or relative changes therein? etc.) However, as a quick check I just ran normalized VIX against normalized SPX (total return), since January 1990, since that’s when Bloomberg’s data for the VIX starts.

"negatively correlated?"

There’s no evident pattern there, but even the most mindless chart junky willing to trust a regression line however inappropriate has to accept that the Reuters has this one wrong — the best fit is a positive correlation.

All the graphs above come from Mathematica, using the Mathematica link to Bloomberg. The graphs with the superimposed pink rectangles were a little challenging to make; here’s the source code if you want to do something similar —

rangesAndScale =
spxDailyLong[[2]][[i + 33]][[
1]]}, (spxDailyLong[[2]][[i + 33]]/spxDailyLong[[2]][[i]])[[
2]] - 1}, {i, 1,
Length[spxDailyLong[[2]]] - 34}], #[[2]] > .0679 &];

ranges = Transpose[rangesAndScale][[1]];

(* we want overlapping rectangles to be unified into a smaller number of wider rectangles, so we need the following *)

unifier[rangeList_] := Module[{i, outp}, outp = {};
For[i = 1, i <= Length[rangeList], i = i + 1, If[i < Length[rangeList], If[AbsoluteTime[rangeList[[i + 1]][[1]]] <= AbsoluteTime[rangeList[[i]][[2]]], outp = Append[ outp, {rangeList[[i]][[1]], rangeList[[i + 1]][[2]]}]; i = i + 1, outp = Append[outp, rangeList[[i]]] ], outp = Append[outp, rangeList[[i]]]] ]; outp ]

unifiedRanges = FixedPoint[unifier[#] &, ranges] ;

legendRects =
Map[{Pink, Opacity[.5],
Rectangle[{AbsoluteTime[#[[1]]], -100}, {AbsoluteTime[#[[2]]],
1600}]} &, unifiedRanges];

DateListPlot[spxDailyLong[[2]], Frame -> {True, True, False, False},
Epilog -> legendRects,
PlotLabel ->
"\"I've never seen a market like this\"\npink=markets just like \
this (6.78% rise in 34 trading days)"]

MySQL in VMWare Fusion

Whatever your development platform of choice, if you work with large amounts of data, you should be running a database management system. In the modern day, high quality DBMSs are available for whatever platform you’re probably using, and some are even available for free. For an important project on which my team is currently working, we have implemented MySQL, in part because it runs on all the same platforms that Mathematica does. Mathematica can interface with it easily, and it comes with decent administration tools. This means that we can program on Windows or Mac and run on either one, transparently.

All of our Macs also have Windows partitions, and we have Mathematica licenses for both. While it is possible to install MySQL under both operating systems, to do so would almost inevitably lead to inconsistencies between the datasets on the two sides. In development, this can complicate debugging, and it’s completely unacceptable in production.

A nice solution is to run the MySQL server under OSX, and to set up the Windows virtual machine as a client to it. This is not hard to do with a virtualization tool like VMWare Fusion, though it is not expressly documented.

To allow the windows account to see the mySQL server on the Mac side, you need to do three things

1. Create permissions on the MySQL server under OSX. For example, create a new account for root, limiting connectivity to hosts matching whatever the windows partition identifies itself as in its network control panel (with wildcards, this often comes to something like win-qae% or win-OFT%).

2. Create a server connection from the client (windows) side. When establishing the connection from windows, you can refer to the Mac by IP address or by its local network name, as appears at the top of the Sharing control panel (Mac-Pro.local in the example below).

sharing control panel

finding the computer's name (also, optionally turning on SSH to allow remote administration)

If you use DHCP, it may be necessary to use a wildcarded version of the laptop’s IP address (e.g. 192.168.%), though this obviously reduces security.

3. In Workbench under Windows, create a connection to a remote MySQL server. This can be pure TCP/IP (which requires you open port 3306 or any other you have set for your MySQL server) or TCP/IP over SSH (which only requires port 22 and encrypts the communication).

This is set up in the Connection Manager (left column, last entry on the WB home screen). Alternatively you can use the New Connection Wizard (click the “New Connection” link in that column).

If you want to be able to administer the database from Windows, including making database backups, you’ll need a couple more steps.

4. Go to the Mac Sharing control panel and enable Remote Login, which turns on SSH.

5. In Workbench under Windows, create a remote “command connection.” This is set up in the Server Instance Manager, which can be reached via the last link in the right column on the workbench home screen. Alternatively you can also use the New Server Instance Wizard.

When you open the server instance manager for an existing server instance, you will see a block of settings titled “Remote Adminstration” or “Remote Management” (select your instance in the left-hand list first). There are the 3 possible type settings (none, win-based, ssh-based) and you need ssh-based for this technique. The SSH hostname is your computer’s name:SSH port number (Mac-Pro.local:22 for example). SSH username is your account name on in OSX.

identify the host for the command connection

set the database connection values

set up remote administration

For the sake of your sanity, do not use the same username for your computer account and your database account. When MySQL asks for the password for user ‘fred’, you need to know which one it’s asking about.

If you have done all these things correctly, it will be possible to query the OSX database from within the Windows partition on the same machine, and to administer the database from either operating system.

Mental defectives, part ii

I did on a previous occasion write about the seemingly powerful relationship between the number of radios in the U.K, and the number of registered mental defectives in that country. In Google books, I recently found a 1958 book titled Principles of Statistical Techniques, which contains examples based on that remarkable dataset, the number of mental defectives in England and Wales.

The passage in question uses the “mental defectives” data from 1935 through 1946 to illustrate the ability to mislead through the use of graphs and charts. The graphs below are courtesy of Mathematica but closely replicate those in the original text.

In[]:= mentalDefectives = {{1935, 86086}, {1936, 88060}, {1938,
92299}, {1939, 99144}, {1940, 101364}, {1941, 100876}, {1942,
98125}, {1943, 98434}, {1944, 99608}, {1945, 102225}, {1946,

In[]:= ListPlot[mentalDefectives, PlotRange -> {86000, 102500},
AxesOrigin -> {1934, 86000},
AxesLabel -> {"Year", "Number of\nmental defectives"},
AspectRatio -> 2, PlotMarkers -> {Automatic, Small}]

tall skinny graph

In[]:= ListPlot[mentalDefectives, PlotRange -> {0, 120000},
AxesOrigin -> {1935, 0},
AxesLabel -> {"Year", "Number of\nmental defectives"},
PlotMarkers -> {Automatic, Small}]

wider graph

As explained in the original text, “The impression obtained from the first graph is that there was a staggering rise in the number of mental defectives in little more than a decade, whereas the second graph gives the impression of a much slower and more gradual increase. These rather different impressions are obtained, of course, by tampering with the horizontal and vertical scales used. . . . it is essential to recognize the importance of supplying graphs, as well as tables, with full and clear labelling and if possible, the source of the information.”

Frankly, their efforts seem pretty basic. We could further steepen the first graph through population adjustment, adjustment for number of doctors, and other techniques.

In googling for more on this remarkable dataset, I found letters to the editor of The British Medical Journal in which one R.A. Gibbons of London, S.W., argues that forced sterilization of mental defectives is not sufficient to protect “the race,” and that it is necessary to sterilize also those people who “from pronounced family history” are certain to “produce idiots.” Dr. Gibbons apparent delivered a paper on this topic to the Obstetrical Section of the Royal Society of Medicine and had in published in the Journal on March 18, 1922. In this letter, he refers to “over 12,000 registered mental defectives for whom [British citizens] are taxed.”

The reference to taxation feels terribly modern (“certainly their plight tugs at the heartstrings, but you just can’t care for everybody and it wouldn’t be fiscally prudent to allow them to run amok.”)

Germany is most famous for eugenics programs, but similar sterilizations of mental defectives were taking place in the United States in the early and mid 20th century. I’ll leave that for another post.