| Seeing Patterns in Data | 2001-12-11 04:27 6 comments |
 by Flemming FunchI'm reading Cryptonomicon by Neal Stephenson. It is sort of a thriller about secret codes and stuff, taking place both in WWII and today. Anyway, one of the recurring themes is about finding patterns in apparent randomness. *Information* is essentially the stuff that is different within a volume of data. If I say AAAAAAAAAAAAAABAAA, there isn't a whole lot of information. It is a bunch of As and a B. The B is what sticks out, of course. There is no information if everything is the same all the time, or if it is just the same repetitive pattern. Information is conveyed when something different takes place, which somebody can make sense out of. Through intelligent analysis one might find patterns of information where the untrained eye wouldn't expect or notice any. An example from the book: a secret installation that is working on breaking codes, using a certain kind of machines, has hired mainly women who are taller than the norm, because of the way the machines are constructed, as they need to be able to reach to the top of them to change the paper. Somebody who had access to personnel records and were analyzing them might notice the height variance. That is information. It points to a deeper pattern. A smart intelligence person would notice that variation and would investigate it further to find out what it meant. The point is that there's a wealth of information to gather from apparently random and incidental data, if one can find the hidden patterns, and the variances from the norm, and the variance from what is truly random. Everything leaves clues. Lots of things are connected with each other, so activities in one place will influence or flavor everything that is even peripherally connected with it. My point is that it is almost a lost field. Particularly when it comes to intelligence gathering outside the field of government espionage and police work. It is a potentially very, very valuable tool, which is being kept for the use by the controlling elite, and little awareness exists of it in the general population. Vocal Authorities in the Scientific Method are key conspirators in keeping this a secret. You will often hear a skeptical scientist defending the world view that the only things that can be said to exist are the things that you can directly and repeatedly prove to exist, to the satisfaction of a group of peers who can repeat the same observations. That makes it very easy to dismiss all sorts of observations of UFOs and paranormal phenomena and stories about conspiracies and foul play, etc. It doesn't really matter how many people saw it, or how detailed and specific the stories are, or how much they back each other up. If the stories don't match the officially sanctioned version of events, and nobody has anything very tangible and undeniable which can be studied - they can be dismissed as false. The criteria for truth for Fundamentalist Scientists is not whether it actually happened, but whether a group of properly credentialed peers will agree that it happened, and that it matches the current theory, and it can be reproduced. That of course doesn't have much to do with what science is really about, which is to explore the universe and arrive at more and more workable truths through experimentation, observation and hypothesizing. We need more of that, and we do need the intelligent rigor that science can provide, in addition to an open mind. Now back to the lost discipline of finding truth in relatively random and incidental data. The point is that you can find out what is going on, even though all your data is indirect. You can find out that there's a fire even though all you have is puffs of smoke. The more different kinds of incidental and indirect data you have, the more you would be able to construct a picture of what is going on, even if you have no direct information. And we're talking about something much more systematic than guessing. For example, take the World Trade Center incident. If you had access to the personnel records of all the companies and agencies in the buildings, and you took a look at who was there and who wasn't there, in what percentages, on the morning of 9/11, patterns would probably emerge. If you looked at changes on or just before that date in many different areas, patterns would emerge. Were there fewer passengers on those planes than normally, or more, or the same? What organizations changed their patterns just before that time. Many conspiracy stories are circulating, claiming to point out some of those patterns, but it is typically only hearsay. Somebody heard that such and such company told their managers to not come to work that day. Might or might not be, but the complete data would show the story. The fact that such data and such analysis usually aren't made available is in itself very suspicious. There are branches of law enforcement and intelligence agencies that know all about analyzing patterns in incidental and indirect data. They just never say anything about it. They leave us with the impression that their investigative techniques are mostly about finding forgotten bags in abandoned cars, and fingerprints and DNA samples and monitored phone calls. All of which are much more a matter of blind chance than the analysis of patterns in the available data is. Somebody mentioned that they could always figure out when the Pentagon was planning something big, as there would be an increase in the pizza orders just before, so all they had to do was to check everyday with the pizza delivery places in Washington D.C. More specific information here. That's the kind of thing I'm talking about, but more different kinds of data sources, and more systematic. And my point is: don't leave advanced analysis of patterns in data to some secret factions of the NSA or the Bank for International Settlements. It is potentially a tool that could provide truth to the masses. Not that I personally can claim much expertise here, but I find the subject compelling. |