Ming the Mechanic:
Seeing Patterns in Data

The NewsLog of Flemming Funch
 Seeing Patterns in Data2001-12-11 04:27
pictureby Flemming Funch

I'm reading Cryptonomicon by Neal Stephenson. It is sort of a thriller about secret codes and stuff, taking place both in WWII and today. Anyway, one of the recurring themes is about finding patterns in apparent randomness. *Information* is essentially the stuff that is different within a volume of data. If I say AAAAAAAAAAAAAABAAA, there isn't a whole lot of information. It is a bunch of As and a B. The B is what sticks out, of course. There is no information if everything is the same all the time, or if it is just the same repetitive pattern. Information is conveyed when something different takes place, which somebody can make sense out of.

Through intelligent analysis one might find patterns of information where the untrained eye wouldn't expect or notice any. An example from the book: a secret installation that is working on breaking codes, using a certain kind of machines, has hired mainly women who are taller than the norm, because of the way the machines are constructed, as they need to be able to reach to the top of them to change the paper. Somebody who had access to personnel records and were analyzing them might notice the height variance. That is information. It points to a deeper pattern. A smart intelligence person would notice that variation and would investigate it further to find out what it meant.

The point is that there's a wealth of information to gather from apparently random and incidental data, if one can find the hidden patterns, and the variances from the norm, and the variance from what is truly random.

Everything leaves clues. Lots of things are connected with each other, so activities in one place will influence or flavor everything that is even peripherally connected with it.

My point is that it is almost a lost field. Particularly when it comes to intelligence gathering outside the field of government espionage and police work. It is a potentially very, very valuable tool, which is being kept for the use by the controlling elite, and little awareness exists of it in the general population.

Vocal Authorities in the Scientific Method are key conspirators in keeping this a secret. You will often hear a skeptical scientist defending the world view that the only things that can be said to exist are the things that you can directly and repeatedly prove to exist, to the satisfaction of a group of peers who can repeat the same observations. That makes it very easy to dismiss all sorts of observations of UFOs and paranormal phenomena and stories about conspiracies and foul play, etc. It doesn't really matter how many people saw it, or how detailed and specific the stories are, or how much they back each other up. If the stories don't match the officially sanctioned version of events, and nobody has anything very tangible and undeniable which can be studied - they can be dismissed as false. The criteria for truth for Fundamentalist Scientists is not whether it actually happened, but whether a group of properly credentialed peers will agree that it happened, and that it matches the current theory, and it can be reproduced.

That of course doesn't have much to do with what science is really about, which is to explore the universe and arrive at more and more workable truths through experimentation, observation and hypothesizing. We need more of that, and we do need the intelligent rigor that science can provide, in addition to an open mind.

Now back to the lost discipline of finding truth in relatively random and incidental data. The point is that you can find out what is going on, even though all your data is indirect. You can find out that there's a fire even though all you have is puffs of smoke.

The more different kinds of incidental and indirect data you have, the more you would be able to construct a picture of what is going on, even if you have no direct information. And we're talking about something much more systematic than guessing.

For example, take the World Trade Center incident. If you had access to the personnel records of all the companies and agencies in the buildings, and you took a look at who was there and who wasn't there, in what percentages, on the morning of 9/11, patterns would probably emerge. If you looked at changes on or just before that date in many different areas, patterns would emerge. Were there fewer passengers on those planes than normally, or more, or the same? What organizations changed their patterns just before that time.

Many conspiracy stories are circulating, claiming to point out some of those patterns, but it is typically only hearsay. Somebody heard that such and such company told their managers to not come to work that day. Might or might not be, but the complete data would show the story.

The fact that such data and such analysis usually aren't made available is in itself very suspicious. There are branches of law enforcement and intelligence agencies that know all about analyzing patterns in incidental and indirect data. They just never say anything about it. They leave us with the impression that their investigative techniques are mostly about finding forgotten bags in abandoned cars, and fingerprints and DNA samples and monitored phone calls. All of which are much more a matter of blind chance than the analysis of patterns in the available data is.

Somebody mentioned that they could always figure out when the Pentagon was planning something big, as there would be an increase in the pizza orders just before, so all they had to do was to check everyday with the pizza delivery places in Washington D.C. More specific information here. That's the kind of thing I'm talking about, but more different kinds of data sources, and more systematic.

And my point is: don't leave advanced analysis of patterns in data to some secret factions of the NSA or the Bank for International Settlements. It is potentially a tool that could provide truth to the masses.

Not that I personally can claim much expertise here, but I find the subject compelling.

[< Back] [Ming the Mechanic]



11 Dec 2001 @ 04:54 by jstarrs : Cryptonomicon
Fascinating post, Ming. I was really balled over by the book when I read it.
One thing related that struck me concerning empiric observation & usage was a very detailed commentary given on the Kalachakra mandala, a very complicated mandala with, I believe, 96 levels.
Whilst describing the number of beams that made up the roof structure, it seemed there was a discrepancy in the numbers given. The commentator replied, "well, according to the tibetans, symmetry is stoopid!'
I now suggest that 'Reading Cracks In The Sidewalk' be made available as a BSC degree.  

11 Dec 2001 @ 06:44 by gyrfalcon : It's 23 a.m.
Yes there are patterns in Data! And this should be your 100 and 23r.d. posting on your web Log Ming! lol...  

11 Dec 2001 @ 07:41 by istvan : Skipping the cracks
Reading this i am beginning to wonder why i newer found that Grail.All my life i have been skipping the cracks on the sidewalk.  

11 Dec 2001 @ 07:46 by bushman : More syncronicity?
Very interesting, and its very correct, I never learned how to spot things till I moved out here to Sedona, at first all I saw was rocks, then one day I found a pottery shard, in the random-ness of red rock. Once that happened it was no problem spotting the pottery shards from then on. That was one patern, now that first piece I found has a drawing, or part of one. It led me to wonder who they where that made this pottery, so soon I was looking for a patern in the patern. I found this secondary patern just 2 days ago. I was accualy looking for something total different at the time using terraserver.com. But as I zoomed out on the satalite pic, I couldn't belive what I saw, it was the drawing on my pottery shard, so I zoomed back in on this image of a canyon system , thats about 10 miles from me. Its exactly the same as the drawing every detail. So I went to the topo map and found that this canyon system contains native American ruins and is called Hanson Ranch, Ive been there and seen the ruins and drawings, but also I know where they moved to now , a piece of the puzzle that the forest service has no clue about, what started as a random find has now snowballed into paterns of major signficance. Im waiting to get a pic back of the pottery shard, it was dated to be 1200 years old by the local natives. I will put the pics up so that I can prove this discovery. One thing is baffleing me to a point about this thing, how did they get high enough to see what thier canyon looked like? My friend fugowe says they used the spirit of the bird to see this. :}
Oh ya, the name of the natives that built the ruins are call the Plakie and the Holokum. The forest service says they where only 4 1/2 ft tall people, and they dated the ruins to be about 1800 years old.  

11 Dec 2001 @ 08:48 by jstarrs : Neat story, Bushman, thanks.
Neat story, Bushman, thanks.  

27 Jan 2002 @ 08:32 by gyrfalcon : 11....

Other stories in
2014-11-07 23:12: Welcome to the 5th dimension
2011-11-07 17:22: Notice the incidental
2010-07-14 13:35: Consciousness of Pattern
2010-06-28 00:03: Pump up the synchronicity
2009-10-29 14:03: Convergent or Divergent
2007-08-05 23:45: Perverse incentives
2007-06-22 22:18: Elementary magic
2007-03-21 14:20: Cymatics and group formation
2007-03-15 01:06: Structural holes
2007-02-27 23:50: Leverage

[< Back] [Ming the Mechanic] [PermaLink]? 

Link to this article as: http://ming.tv/flemming2.php/__show_article/_a000010-000050.htm
Main Page: ming.tv