The Cats Were There First

A common way to imagine the future of physical AI is as a sequence of increasingly difficult environments.

First the factory.

Then the warehouse.

Later the laboratory or hospital.

Eventually the home.

The progression seems reasonable. A factory is structured. A home is not. Industrial robots can operate inside carefully defined boundaries, while a robot in a home must cope with chairs that move, doors left open, objects on the floor, people walking through its path, animals appearing unexpectedly, and circumstances that were never anticipated by its designers.

But perhaps this way of arranging the problem begins from the wrong direction.

Nature does not wait until an animal possesses a complete model of its future environment before placing it into the world.

It does almost the opposite.

The animal arrives with constraints, capabilities, senses and mechanisms for learning.

Then the world teaches it.

Birth constraints

Imagine bringing a new AI robot into a house for the first time.

Instead of supplying it with an enormous prefabricated description of houses, furniture, human routines and every conceivable domestic situation, give it something closer to birth constraints.

Do not collide with people.

Maintain balance.

Preserve sufficient energy.

Avoid damaging yourself or the environment.

Attend to movement.

Distinguish what appears fixed from what can move.

Remember consequences.

Explore when uncertainty is high.

Revise expectations when reality disagrees.

The robot does not initially know this house.

That is the point.

It has been given enough structure to begin discovering one.

This changes the engineering question.

Instead of asking:

we can ask:

Those are very different problems.

Put a cat in the house

Consider a cat entering an unfamiliar home.

The cat has never seen this particular arrangement of rooms, chairs, cupboards, stairs and doors. Yet it begins acquiring useful knowledge almost immediately.

Where can I go?

Where can I hide?

Where can I hide and not be seen?

What can I jump onto?

What can I get underneath?

Which openings are too small?

Where can I observe the room safely?

Which routes allow escape?

Where do humans appear from?

Where is food?

The cat does not need a CAD drawing of the building.

It moves.

And movement changes what can be known.

Something hidden behind a chair becomes visible when the cat changes position.

A high surface becomes reachable after another surface is discovered between the floor and it.

A narrow opening becomes classified not by its name but by attempting to pass through it.

Movement is therefore not merely an output of intelligence.

The organism acts upon the world partly in order to discover what the world is.

The world supplies the data

This leads to a different interpretation of the frequently stated physical-AI problem that there is insufficient training data.

Perhaps there is insufficient prefabricated data.

That does not necessarily mean there is insufficient data.

The physical world is producing differences continuously.

A robot capable of structured exploration can recover some of the information it requires simply by interacting with its surroundings.

Approach an object.

Observe it from another direction.

Touch it.

Apply a small force.

Observe the response.

Attempt a route.

Encounter a constraint.

Try another route.

The resulting sequence is not merely sensor data.

It is a relationship:

The robot can therefore perform experiments.

A static training corpus tells the robot what happened when somebody else encountered the world.

An embodied learning system can ask the world another question.

And the world can answer differently.

The library is outside

This connects physical AI to a more general principle.

The world contains far more information than an individual system can sensibly internalize.

A house contains enormous physical detail. Surface textures, friction, illumination, acoustic properties, object positions, temperatures, mechanical tolerances and constantly changing configurations could be represented at increasingly ridiculous levels of precision.

There is little reason to reproduce all of it internally.

The detail is already there.

What the robot needs internally is sufficient calibration and indexing to recover relevant detail when it becomes necessary.

When crossing an empty room, coarse geometry may be sufficient.

When approaching a glass, relevant detail expands.

Where is its edge?

How wide is it?

Is it moving?

How firmly can it be grasped?

Is the surface slipping?

What force is being returned through the fingers?

After the glass has been safely placed on the table, much of that momentary high-resolution information may no longer matter.

The world can be read again when necessary.

Intelligence therefore may depend less upon carrying an exhaustive representation of reality and more upon knowing where and how to look again.

Memory should not become another warehouse

There is a trap here.

A robot equipped with cameras, microphones, tactile sensors, force sensors and position sensors could generate staggering quantities of data merely by existing.

Keeping everything would reproduce internally the same problem we were trying to avoid.

Learning therefore requires forgetting.

Or, more precisely, compression.

Repeated observations can progressively become structure:

After passing the same sofa five hundred times, the robot does not necessarily need five hundred complete visual recordings of the sofa.

It may need something closer to a relational memory:

sofa — usually here

sofa — large obstacle

underneath — partly accessible

behind — possible space

humans — frequently sit here

position — may occasionally change

Even this description is unnecessarily linguistic.

Internally the information may simply exist as relationships between states, actions and consequences.

The important component is the index.

The robot need not memorize the library.

It can build a catalogue.

When reality contradicts the catalogue, attention expands again.

Prediction fails.

The robot looks again.

Memory is revised.

The world has corrected the model.

A chair moves

Suppose someone moves a chair overnight.

A system attempting to maintain a perfect internal replica of the home now contains an error.

But most of what the robot learned about the chair has not become false.

The chair can still support weight.

It can still obstruct a route.

It can still be moved.

There may still be space beneath it.

Only some relationships have changed.

Its location has changed.

Perhaps the available routes around the table have changed.

A compressed relational representation permits local revision without requiring the world to be learned again.

This suggests that useful memory should separate invariants from circumstances.

The particular position of a chair is temporary.

The consequences of encountering chair-like structures may be much more persistent.

Learning is partly the process of discovering which is which.

And this introduces another useful distinction.

The story may say:

the chair was moved during the night.

But the robot does not encounter the story first.

It encounters a changed field.

A route is no longer open.

An expected visual edge appears elsewhere.

A previously available passage has disappeared.

The story is reconstructed afterwards.

Rabbit hole: What is an object?

At this point even the apparently simple concept of an object becomes questionable.

To a conventional recognition system:

But to an animal—or perhaps to a sufficiently ecological robot—the chair is simultaneously a collection of possibilities and constraints.

It can be walked around.

It can be sat upon.

It can block a route.

Something can hide beneath it.

Something can be placed upon it.

It may move if sufficient force is applied.

Its meaning depends partly upon what an agent can do with it.

This is close to the ecological concept of an affordance.

The world is not merely composed of named objects.

It is composed of opportunities and refusals.

The opening permits passage—or refuses it.

The surface supports weight—or refuses it.

The object moves under force—or refuses to move.

The route remains open—or becomes blocked.

Behavior can emerge from discovering these relationships without first assigning words to them.

The boundary does not need to explain itself.

The attempted continuation either survives or it does not.

Then language arrives

Now something curious happens.

The robot has already learned physical relationships.

One object can be above another.

Something can disappear behind something else.

An object can be inside a container.

The robot itself can move upward, downward, forward and backward.

It can stop.

It can stand.

Perhaps it can sit.

Then humans begin making recurring sounds while these relations occur.

“Up.”

“Down.”

“Behind.”

“Under.”

“Sit.”

The robot does not necessarily need a dictionary definition of behind.

It already knows behind.

It has encountered the spatial relationship repeatedly through movement, visibility and occlusion.

What it must discover is that the people around it use a particular sound pattern to index that already-known relationship.

Language can therefore emerge as another indexing system.

The sequence becomes:

rather than:

Put the same robot into a Swedish-speaking home and the labels become upp, ner, bakom and sitt.

Put it into an English-speaking home and they become up, down, behind and sit.

The physical field has not changed.

The linguistic indexing of the field has.

The robot need not learn what behind means from language.

It can learn behind from the world, and later discover what the world around it calls it.

Rabbit hole: Spatial language

This suggests that language itself may have an ecological foundation.

Before a word such as under can be useful, there is already a physical relationship to which it refers.

Before inside, there is enclosure.

Before near, there is distance.

Before move, there is state change.

Before stop, there is the transition from movement to nonmovement.

Language can compress these recurring structures into symbols that allow one organism to direct another organism’s attention.

A word may therefore function partly as an index into an already existing field of experience.

This reverses a common model of language learning.

We do not necessarily need to begin with:

We can begin with:

The robot enters an already-existing household

Now change the experiment slightly.

Do not construct a special training environment for the robot.

Do not bring in an animal as a specimen.

Bring the robot into an ordinary household.

The household was already happening.

The people were already living there.

The chairs were already being moved.

Doors were already being left open.

Meals were already being prepared.

Objects were already being misplaced.

And the cats were already there.

This order matters.

A training dataset is prepared for the learner. Its examples exist because somebody decided that the learner should encounter them.

The household has not been prepared for the robot.

It exists for its own inhabitants.

The robot is the newcomer.

And there are two cats

One cat is useful.

Two cats reveal something more.

Something interesting does not only happen between the robot and a cat.

Something happens between the cats.

One initiates.

The other declines.

One disappears around a corner.

The other waits.

A chase begins in the corridor.

It ends beneath an armchair.

One cat occupies a passage the other intended to use.

A paw appears from beneath the sofa.

A previously sleeping animal suddenly becomes an accelerating obstacle.

None of this requires the robot.

The robot did not initiate the event.

It did not request a sample.

It did not choose its timing.

It may not even have been noticed.

The event belongs to a coupling between two other autonomous beings.

The robot only witnesses it.

This changes the meaning of the outside library.

The world does not merely contain information outside the learner.

A household is not silent between queries.

It has its own event stream—autonomous, unscheduled and independent of the robot.

The learner is not its author.

Two observational paths

Two cats also provide another useful property.

The same physical environment is being explored simultaneously from different positions, with different intentions and different consequences.

One cat passes beneath the chair.

The second goes around it.

One notices movement behind the curtain.

The other does not.

One approaches the robot.

The other observes the approach from across the room.

The difference between those paths becomes observable.

This is not because either cat was instructed to perform an experiment.

It emerges because autonomous agents occupy different relationships to the same field.

Difference itself becomes information.

The cat is learning too

At first a cat observes the robot.

Does it chase me?

Can it see me here?

Does it make noise before moving?

Can I walk behind it?

What happens if I stand in its path?

Can I sit on it?

The robot changes its behavior because of the cat.

The cat observes the change.

The cat changes its behavior.

The robot observes that change.

The second cat observes both.

Now we no longer have:

We have something closer to:

But even this diagram is misleading if read as a serial chain.

All of these relations can exist at once.

The field changes locally.

Those changes alter other available relations.

The ecology responds.

This is no longer adequately described as a training dataset.

It is an ecological system.

A dataset cannot play with you

This exposes an important limitation of prefabricated data.

A dataset contains examples.

An ecology produces new ones.

A prerecorded sequence cannot notice that the robot learned yesterday’s problem and behave differently today.

Another adaptive organism can.

The robot predicts the cat.

The cat observes the robot.

The cat changes direction.

The prediction fails.

The robot updates.

Failure is now information.

But the cat has not failed.

The robot’s model has.

That distinction matters.

The other organism is not noise merely because it declines to conform to the learner’s expectation.

Its independence is precisely what makes the event informative.

A cat may approach.

Move away.

Ignore.

Interrupt.

Play.

Sleep.

Refuse.

These are not six commands in a curriculum.

They are six things an autonomous being may do.

Some permit continuation.

Some redirect it.

Some terminate it.

Some begin another interaction entirely.

Over time the differences reveal boundaries.

Not because the cat has been instructed to teach boundaries, but because autonomous behavior has consequences.

A sleeping cat is not missing data

Sleep deserves special attention.

It is easy to misread.

A machine-learning reflex might describe a sleeping cat as missing data—a period during which useful behavioral samples are unavailable.

But the cat is not missing.

It may still be lying two metres from the robot.

Its spatial position remains available.

Its body still occupies space.

What has changed is the relational field.

The set of available interactions has contracted.

The cat has become unavailable on its own terms.

This is not missing data.

The unavailability is not caused by the learner.

It is not negotiated with the learner.

It is not scheduled for the learner.

A dataset does not do this.

A dataset waits.

A living system does not have to.

The robot must therefore learn something subtler than presence or absence.

An entity can remain present while the relations available to it change radically.

The constituents remain.

The configuration changes.

The possibilities change with it.

The curriculum was never issued

In the first version of this thought experiment it is tempting to say:

The GPUs perform the arithmetic. The cats provide the curriculum.

But that is not quite right.

The cats provide no curriculum.

They have their own lives.

The curriculum is reconstructed afterwards by the learner.

The autonomous beings arrived first.

Their movements, hesitations, approaches, refusals, disappearances, games and sleeps occurred because those beings were doing what they were doing.

The robot observes consequences.

Recurring relations accumulate.

Some survive repeated encounters.

Others collapse.

From those differences the robot constructs useful structure.

The world was already happening.

The curriculum came later.

From environment to ecology

This may be the more important conceptual change.

A home should not be regarded merely as an environment into which a finished AI system is deployed.

It is an ecology containing adaptive agents, persistent structures, temporary arrangements, routines, exceptions, language, consequences and continual change.

The robot becomes another participant.

Humans adapt to it.

Animals adapt to it.

It adapts to them.

Animals adapt to one another regardless of it.

Even the physical environment changes partly because of those interactions.

Learning therefore does not naturally follow:

It becomes something closer to:

There may never be a final trained state.

And perhaps there should not be.

A final trained state would imply that the world has stopped producing relevant novelty.

The household has made no such promise.

Rabbit hole: Re-reading reality

This produces an unexpected connection to re-reading.

A stored document can be read again because information remains available outside the current act of interpretation.

The physical world offers something analogous.

When the robot’s compressed model becomes insufficient, it can return attention to reality.

Look again.

Move again.

Touch again.

Test again.

But physical re-reading contains an additional complication.

The page may remain approximately the same between readings.

The household does not.

The chair may have moved.

The light has changed.

One cat is asleep.

The other is now behind the door.

The human who was sitting has stood up.

Re-reading reality therefore does not mean recovering an identical sample.

It means returning to the source.

The source is allowed to have changed.

That is not a defect in the method.

The change may be the information.

The world has not been exhausted by the first observation.

Another encounter with an object from another position or under another condition can expose another relationship.

The model is provisional.

The external source remains available.

Or unavailable.

Or changed.

Again:

the library is outside.

Nature teaches partly by refusal

There is another consequence.

Learning does not require the world to explain why an expectation failed.

The robot attempts a route.

The route is blocked.

It applies a small force.

The object does not move.

It expects the cat to continue forward.

The cat turns away.

It attempts an interaction.

The sleeping cat does nothing.

The world has not supplied a verbal correction.

It has supplied a boundary.

Some possibilities survive contact with reality.

Others do not.

Repeated refusals begin to expose structure.

The robot need not initially know why.

It can first learn the contour.

Explanation can come later.

The five phases reconsidered

Industry forecasts often arrange physical AI according to environments of increasing deployment difficulty:

factory first, then warehouses and specialized applications, later regulated complex environments, and finally the general-purpose home robot.

As a commercial deployment forecast, such a sequence may be reasonable.

But it need not describe the natural architecture of learning.

Nature does not appear to solve the forest before releasing the squirrel.

It releases an organism possessing sufficient constraints and capabilities to begin learning forests.

Perhaps physical AI should borrow more heavily from that architecture.

Instead of engineering every possible behavior before deployment, engineer the capacity for safe ecological learning.

Instead of attempting to internalize every possible home, create a system capable of learning the home in which it actually finds itself.

Instead of treating novelty solely as an error condition, allow novelty to generate information.

Instead of making memory an archive of everything experienced, compress experience into useful invariants and indexes.

Instead of treating language as the beginning of meaning, allow language to attach itself to relationships already discovered in the world.

And instead of imagining the learner as the centre of an environment built for its education, place it carefully into a world that was already there.