However, should the universe itself be mathematical (which I believe is very, very likely), then the monkeys and their maneur throwing behaviors are just as mathematical as the logic of Wittgenstein, just expressed differently.

Perhaps the rigidity of some of the academics regarding the universality of mathematics is to blame for current generation’s penchant for glorifying ignorance. What people need to realize is that they are already doing math the moment they are born into the world, and the study of the discipline of mathematics is merely a means to provide lucid understanding of the behavior we and everything else in this universe already indulge in.

On the other side, math itself follows from our ability to abstract, is is a special way of understanding things, and in no way the essence of being human.

As someone said, there are humanities, there are natural sciences, and there is math - the only supernatural science.

“There is no such thing as a post-human society that any humane person would not find abhorrent.”
Yeah, yeah, and dogs don’t like living in houses so I guess houses are out too.
Your flaw is in thinking that humans are at the pinnacle of something and that we are the ultimate judges of good.

I think math is more a language of logic.

And here you describe every dystopian novel ever written: the concept that humanity can somehow be reconciled with reason. There is no such thing as a post-human society that any humane person would not find abhorrent.

Go read Russell’s Principia then Godel, and tell me that mathematics isn’t subject to the same hubris and tragedy as every other field of human endeavor.

Often this utility is more-or-less imposed by your body. Three billion years of hungry ancestors demand the chocolate and you are powerless to resist. Overriding the rational decision here is hardly self-willed!

However to demonstrate the sort of will I think you’re referring to, you wouldn’t have to override the rational decision but the utility decision. Say you really wanted some food and had no good reason not to buy it (suppose it’s allowed by your diet). You have enough money. Instead you decide to throw the money on the ground and go hungry. Why on earth would you do such a thing? Isn’t it obviously self defeating and a loss with no gain? You could do that to prove your self-will, but that would be serving a gain (proving something you wanted proved). But if you didn’t even want to prove that, if you didn’t have any reason at all (good, bad or crazy), would you do such a thing? No, of course not. Deterministically not.

The more you knew, the more decision-branches would come to seem like throwing your money on the ground: a pure loss. You deterministically wouldn’t do any of those things — and so your choices would be narrower.

Try actually writing a program to implement what you’re talking about; it’d be a useful exercise.

You may want to start with “how does my AGI solve narrow problems that are already solved?” - say, for example, how does it learn to play chess or draughts given the rules of the game? or “how would my program understand natural language in a microdomain like SHRDLU?”

*************************

It’s relatively easy (albeit time-consuming) to come up with a particular mechanism for solving a specific problem, or narrowly defined problem class, which I might call P. But usually that particular mechanism will fail miserably on other problems, say on Q and R. If you then write programs that solve Q and R well, you end up with three programs which are written in totally different styles, perhaps even different languages, and which have no obvious common generalization.

This is the state of current AI research. Lots of disparate problem-solvers which work well on very specific problems, but break absolutely outside these narrow domains. Commonsense knowledge and natural language understanding remain elusive. No-one knows how to write a program that will solve problems that the designers hadn’t thought of; thus current AIs cannot understand something that their designers didn’t already understand, although AIs can solve specific instances that their designers would not have been able to solve.