| Incidental Data Analysis | 2004-05-05 07:13 12 comments |
 by Flemming FunchThere's one discipline I'd like to be an expert in. Well, there are many subjects and disciplines I'd like to explore, but that I don't get around to. I'm, however, very intrigued about techniques for figuring things out without direct access to conclusive source data. Everything leads traces. Everything that happens leaves some kind of signs in the environment. Nothing real is completely isolated from its surroundings. Every event interacts with its circumstances, and will influence them in ways that can be noticed. Even if you don't have the thing itself, you will always have access to something that was in contact with it. If you're skilled enough, you can find things out indirectly that aren't available directly. A good example of the principle is the rumor that whenever the Pentagon is planning a big operation, the number of pizzas ordered at night from Washington D.C. pizzarias to be delivered to the Pentagon, will increase. Obviously because a bunch of people are working long nights there, and they're hungry. It is kind of obvious. If you can find out where they order their pizzas, and you watch the graph of how many pizzas are ordered at which times, it can tell you that something is going on. That also brings up the obvious objection to such an intelligence method. It doesn't really prove anything. It doesn't at first tell you *what* is going on. Maybe they're just doing the yearly audit in accounting. Maybe they just changed pizzarias. Maybe their own kitchen is out of order. A lot of activities can stay very hidden exactly because such approaches generally aren't officially given much credence, and there are few organized techniques available to the general public for applying them. Which is in part because those people who're doing big things and have big things to hide would prefer that nobody can figure them out easily. And because they use such techniques themselves. Pattern matching and profiling is one angle of it. If you find the data that sticks out, that doesn't match the normal profile of how something behaves, or that matches a profile of particular kinds of unusual behavor - you know something is going on. Again, you might not know exactly what, but you know where to look. Like how your bank might freeze your credit card because you did something they considered unusual, like buying jewelry in Hong Kong, when normally you only buy groceries in Wisconsin. But they're rather bad at this science, so they make mistakes half the time. The folks who control large amounts of centralized data have a leg up on everybody else, of course. If one has access to your bank records, your telephone records, your travel records, etc, one could say a whole lot about you. I suspect the FBI and CIA folks are not very good at it, though, and they're probably just drowning in data that they don't know what to do with. However, I also suspect that there are some groups that are experts at analyzing patterns in such data, but they probably aren't talking about it. Anyway, the working theory is that any kind of incidental data can lead you to figure out big things that are hidden. Maybe not maliciously hidden, maybe just obscured from view and not yet discovered. If I stood down on the corner and catalogued all cars that drove by, and I did that for a while, I'd learn something from the patterns. Duh, yeah, how many cars drive by, of course, which is useful for traffic planning. But if I look a little broader, I might see surprising data. Why are there an unsual number of red cars driving by every day around 3 o'clock? That might not provide the answer, but it might tell me what to look for next. Doesn't have to be anything exciting. I might just learn that they're company cars, and that the sales people for a certain company meet for donuts in a certain place at that time. But there's something to find out. We're generally being sold a picture of the world as being very confusing and disjointed. A lot of discrete events that you can't all keep track of, and the only way of making sense of it is to listen to somebody's two minute summary on the news, or by adopting some ideology that summarizes the world in simple ways I can just believe in without examining it too closely. But I don't buy it. I think there are better ways of making much better sense of even a world represented by huge amounts of data. Obvious techniques are to count things, and to categorize and catalogue them. And cross-relate different kinds of data. And then to look for certain patterns of things being "wrong" or "right" or "off" or "on" about them. E.g. things being out of sequence, misplaced, misnamed, or things fitting unusually well together, or working unusually well. If a hundred carpenters seem to produce a certain typical number of chairs and tables, and one of them produces twice as many, it tells us something. We might discover that he just works twice as long, or we might discover that he has an approach that works better. If a certain news agency produces considerably more erroneous news stories than the other news agencies, there's a story there somewhere. Incidental data usually isn't random, even if it looks like it at first. Really, most things in the world are connected with each other, so things usually are the way they are because of the way other things are, and there's a bigger picture there. That also opens the door for apparently non-sensical ways of divining what is going on. If Uncle Joe's left knee hurts before it is going to rain, and that is reliable, we don't really have to know exactly how that comes about. An analysis of his success rate is all we'd need. Likewise, if somebody has a system of divination based on tea leaves, and it happens to work, that is valid data as well. Even if somebody else would like you to believe that you can't say anything useful about things you can't explain as a direct cause-effect relationships. There's a certain centralized power structure that exists in terms of information. Those who appear to hold the centralized stores of the "proper" data, like governments, the police, banks, credit bureaus, media companies, scientists and educational institutions, pretend that they have a monopoly on telling you what is going on. Where I postulate that we've gotten to a point where grassroot networks of regular folks easily could be as informed as any of those. But not just by passing vague rumors and opinions around. Some tools and proven disciplines would help. Nothing happens in a vacuum. Life leaves tracks. Almost any kind of incidental data source, if analyzed a bit, can tell you where there are tracks. By combining a sufficient number of dimensions of data, you can probably see where the tracks lead, and you can make out the silhouette of what is there. |