I’ve come a long way, baby

It has been a few months since I posted here, and surprisingly much water has gone under the bridge. For most of my enlightenment, I owe Eliezer Yudkowsky, who is much smarter than me and something of a hero. (He’s also an autodidact. I will catch up to you!)

The two largest changes: I have come to understand a bit more of the Bayesian ideal of reasoning from E.Y.’s posts on Overcoming Bias blog, and I have reached a much better idea of artificial and natural intelligence (much credit to this paper), which has allowed me to unify my treatment of the two. Intelligence is that which hits a small goal in a large search space. General intelligence does this in the general case.

This has allowed me to understand that humans are not (as we arrogantly presume) great shakes in the generality department. You can see human generality in the understanding of maths, which is an easy problem from any objective standpoint. It’s rare and mostly weak. We are almost specific intelligences, as tailored for being a monkey as a chess program is for its niche.

This kind of thing has led me to four, if not new then at least newly emphasized positions:

1. I am now a transhumanist. I want to do better than this meat mind. It is suited to apes, not to people. As E.Y. puts it: how strange we are, a mind with the form of evolution. The bridge between the era of evolution and the era of mind. “Artificial Intelligence” should be read as meaning “mind born of mind”. The artificial is a badge of honor!
2. I am now a singularitarian. I understand the power of self-improving AI. Only two events are comparable: the big bang and the first ancestor of life. If we design an AI that is unsafe, we will be as chaff before the wind. Most kinds of unsafe AI are also boring (a variant of sorcerer’s apprentice mode). Making interesting AI will be hard. Making safe AI will be very hard. Technical trends that make it easier to build unsafe AI are a danger, not a blessing.
3. I am now an atheist fully. I understand mind, and I can’t harbor any romantic illusions about its nature. Mind is mathematical and computational. These are scientifically tractable concepts. Being info-computation is actually more interesting than being some undefined “soul”. Information has properties, and they are interesting ones.
4. Not coincidentally, I really like maths!

Addendum to the below: E.Y. has convinced me that we can’t dare to build a neural AI, because it’s mathematically intractable and can’t be proven safe. The same has to go for an instinct-simulating AI, except to the extent instinct approximates Bayesian reasoning. Although I can’t yet follow the maths, I am convinced on good authority that any system of understanding either maps to Bayesian probability, or is inconsistent — and that it only has value at all to the extent it approximates Bayes’ law. Ergo, any good AI will either be Bayesian, or a hack. (Humans are a hack.)

And now for something completely different. Some random thoughts that interested me:

• Does conventional AI go about modeling natural language in the wrong way? Suppose: prose is-a poem. Poetry is the superset!
• E.Y. explains that evolution has a mathematical upper bound on the information it can sustain against mutation. Can we apply this to the quasi-evolution of institutional memory in the economy? (The kind of information I’m talking about here is embodied in structure, not personal knowledge inside people’s heads. That’s why I think it will behave differently to intelligence.) Is there an upper bound on the information that corporate capitalism can sustain against economic churn? Suppose there is a limit which is predicated on dumb humans. Can we act as smart humans by designing an information pump that re-inserts this lost information? Can we design company structures that would be amenable to this sort of on-the-fly rebuilding?