HOW TO BUILD THE AI

Root Goal: The Good Depends on Goals

Good and bad are not Platonic ideals. They are not things existing on their own in the world. Instead, good and bad only exist in relation to goals.

Something is right, in the context of a particular goal, if it contributes to the ultimate achievement of that goal. Something is wrong if it hinders achievement of the goal.

Goals themselves can be right or wrong, of course, since practically every goal is a sub-goal leading to some higher goal. So a thing might be right in the narrow context of one goal and at the same time wrong in a wider context, if that thing's immediate goal is wrong.

A goal can be wrong in two different ways:

  1. It can fail to contribute to achieving its next-higher goal. Thus, a sub-goal can be wrong in the context of its higher goal.

  2. It can fail to contribute to the highest goal of all. Even if a goal really contributes to achieving the next higher goal in the hierarchy, that next higher goal might not contribute to achieving the root goal.

Example 1: A boy runs to get his squirt gun to put out a fire. Getting the squirt gun is a bad goal because it won't significantly help in putting out the fire. The boy should get the garden hose instead.

Example 2: If you are trying to murder someone, it's right to load your gun with real bullets. On the other hand, it's ultimately wrong to load your gun with real bullets, because it's wrong to murder someone in the first place.