Directions in Object-Oriented Programming From Actions to Agents

in the timeframe of 15 or 20 years ago Alan Kay conceived of the Dynabook which

most of you probably know was a

conceptual forerunner of the PCs and and and probably some future machines that

we've not not even seen commercially yet Alan Kay obviously as the man who's

always been ahead of his time he was the

inventor of the first small talk some of you may know that there were actually

several small talks the one that you now know a small talk ad was the fourth or

fifth in a sequence of small talks Alan Kay did that first that very first small talk and currently he is a an Apple

fellow and prior to that he was a chief scientist at Atari with credentials like

that I found it interesting to note in his biography that the only professional

Society of which he is a member is the International Society of organ builders

without taking any more of Alan's time let me now bring up Alan Kay

[Applause]

I have a piece

oops

so I found it necessary when giving talks in the East Coast to put the slide

up so that people who might be offended can leave early I thought this might be

an especially appropriate place well I

think a good way to start off is to

recall a statement by William James made

about a century ago about the three stages in the life cycle of any idea and

the first stage is that the new idea is condemned as being the work of madmen

then a few years later it's remarked on as being commonplace and obvious all the

time and the final stage is when the original condemned errs claim to have invented it

I think we're at that stage now with

objects in window oriented and pointing user interfaces which to me means that

they must be hopelessly obsolete so

we're actually we're actually having a symposium here about a patient who's

already dead but in the grand tradition

of computer science will live on for at least a century this is gonna be a

different talk than norms talk norm is a norms talk as a practical talk by a

person who got interested in

object-oriented programming as a way of implementing large-scale complex systems

in the context of a university where you

don't have a lot of people to do the

work and a lot of the work is transient and this this need to deal with

complexity in a way is one of the main selling points in object-oriented

systems and people have this notion that

because object-oriented systems are usually worked on by small groups that

it's something mainly for small groups the main reason small groups use it I

think is because it's one of the few vehicles that a small group can do truly

interesting things in a short time but if object-oriented systems are

implemented in a in a thoughtful way than by their very nature and we'll talk

about this in a bit if the protection mechanisms that are inherent in the

notion of object-oriented programming are also implemented which they often aren't then one should be able to

organize very large-scale projects as well because the whole joy in the

object-oriented design is the notion that when you put an object in the

system it should be able to make use of what's there but its presence in the

system shouldn't disturb any other objects that's in the system it should

be something that is self-contained it should not be able to be destroyed by

the system and so forth and so the

in an object-oriented system that's well designed you should be able to add new

knowledge in an incremental fashion almost linear fashion how we can a

little bit later we'll talk about a couple of the places where that isn't really true and maybe some of the

solutions for it but I thought a good way of starting off would be to just

take a look at a couple of object-oriented systems of the past or

almost object-oriented systems the first one and I understand they're only a few

people here last night to look at it I have an excerpt maybe this is even a

different film of Ivan Sutherlands sketchpad which was one of the earliest

object-oriented systems although Ivan didn't really quite think of it that way

there's an interesting history behind it so let's show that first tape this is

implemented by making this room that's -

the last computer in the u.s. to have its own roof this is what you saw there

was the very first implementation of the rubberband technique and here's something you don't often see even in

graphic systems today he pointed he did a rough sketch he pointed to those edges

and told him he wanted them all mutually perpendicular and sketchpad figured out

how to do it we just saw that sketchpad was also the first system to have a

window this is all done in 1962 there he

pointed to those two lines and told them that he wanted them to be parallel and

now the constraint is to be collinear so

these little dashes will be precisely lined up over the lines underneath this

is the first time when people discovered that the light pin was not a good input device

and in spite of that it's been reinvented at least 90 times in the last

25 years so he's made a hole in the

flange and now he wants a rivet and

again the whole idea behind sketchpad is you should be able to quickly sketch in

an approximation of what you wanted who's going to use that as the center

for the arc and then get the system to complete the drawing by implementing

various constraints or rules and here he's saying make those four lines

mutually perpendicular the doing of that

pulls the diagonal line which pulls the center which pulls the arc which makes a

perfect rivet so this is also the first

non procedural programming system

showing that no matter what you do and of course he could have added further

constraints that would constrain the ratios of those sides so that no matter

how you distorted it always would come back to something that was the same

relative size so that was a master what

we call a class today and here's an instance of that rivet the instances can

be scaled and rotated in real time you might wonder why is a display jumping

around like that well I didn't display

is back then the tx2 here is putting up every screen individually in fact memory

for each separate dot so this is a software oriented display

and when you put up a lot of dots that starts flashing because it's tracking

the pen here are a couple more instances

listen you might imagine led to the development of better displays now he

doesn't like the cross piece on the rivet so he goes back to the master and

says make those invisible they're still there but they're invisible now and lo

and behold we see that the instances feel that change dynamically and

remarkably sketchpad even had a form of inheritance who's actually one of the

first object-oriented software systems so now he's made this constructed thing

into a master and now he can get instances of it I once asked Ivan

Sutherland many years ago how is it possible for you to in a single year of

1962 to do the first computer graphics system the first non procedural

programming system and the first object-oriented software system and he said well I didn't know it was hard ok

stopped them made a tremendous

impression I had to learn it that's 22

now when I first went to graduate school

the desk at the University of Utah Dave Evans had one of these theses in and

hand out and you had to prove that you understood it before you're allowed to sit down and think I might

this isn't worthwhile looking at in any detail maybe I'll just use this from the

main line here this this is essentially

the pathway that I took getting interested in this stuff I was in the

Air Force and there was a file system on the Burroughs 220 that was what we would

call today object oriented back in those days I didn't have operating systems of

course and these days did not have operating systems either back then there

was more excuse for it

and the Meniere's had the problem of how

can I hand data tapes around Air Training Command

they all have Burroughs 220s but there is no standard operating system and so

somebody some genius in the late 50s came up with the following idea he said

well what if we had to take format like this with each long tape record is going

to be made up of three parts the first part is going to be a relative transfer

vector into the second part which is going to be a bunch of Burroughs 220

code that 220 code will know how to

interpret the third part which are going to be our unit records so all you have

to do when you get one of these tapes is you simply read the transfer back to the

front part of it into a special place in core and then you can do anything you

want with these tapes because we'll have a standard protocol in this transfer

vector that's the way that file system worked back then heck of a good idea I

was just mentioning to somebody this morning that the first time I ever did

what we would call object-oriented programming today was done on the 1401

in Auto coder because we used this technique on the 1401 s there as well

and made up macros to allow us this generic access to things

terrific idea why didn't it last well COBOL actually drove it out the adoption

of COBOL as a as a way of getting ahead

actually gave us a weaker metaphor that we're still saddled with today

the borough's be 5,000 actually implemented these ideas and hardware it

it antedate sketchpad and probably sent seven out of the top ten ideas ever

thought up in software are actually implementing the hardware of this machine Burroughs made the enormous

mistake of trying to explain this machine to their customers back then

they actually hired college graduates and taught them how the machine works so they could go out and tell the customers

how good it really was and I was I think it was the third machine that I learned

back then and the these college graduates scared the heck out of all of

these old data processing managers who are just struggling out of punch cards

and the machine was late as well so it just never caught on sketchpad and then

Simula was the first programming

language that I would call trying to be

an object-oriented language and there are two versions of it in between I

designed a language called flex which is

an attempt to simplify Simula and in some sense some real sense the small

talks were an attempt to send to simplify Simula as well but there are

all these other things that fed into it over here which the most important ones

for I think object-oriented programming are the Lisp influences and over on this

side influences that are basically either philosophical or coming out of

cell biology now reason I bring these

these up is because I think it's it's

critical when thinking about object-oriented programming not to look at the the systems that are out there

today and think this is object-oriented programming because in my opinion it

isn't to me small talk eighty is not an object-oriented language because it

lacks one of the things that I thought was most important in object-oriented

design and that is differential protection which means that you should

be it when an object is put out into the world that should be able to protect

itself against various cendars so for instance if you have a

file object but you would like the file object to be able to do is to

discriminate about who's sending it the message to change bite number eight in

the file and it's possible to do that in

small talk but it's not set up to actually do that you can act in small

talk you have to instrument every method that you want to protect in order to in

order for it to do something the earliest small talk it was combined

with what's known as capability protection there was objects were seen

as a generalization of capability protection and that protection was built

in so that no object could command another one to me the biggest difference

between whether you really have an object-oriented system or not is whether you are really sending messages or not

and small talk and Objective C and C++

and all this you aren't really sending messages you're actually just doing subroutine calls which means that in

some sense the sender guy on the outside is commanding the object to do things

and the whole notion is not for doing

commands but for doing something else let me give you a sort of a feeling for

that

when I started thinking about this stuff

for real after I done a done a few

failures I came to realize that the only

model that I knew of either done by man or in nature that really could handle

complexity in any appreciable fashion was a biological one the biological one

ninety percent of a cell's effort is going into its own protection only ten

percent of the cells effort is going into contribute or less as going into contributing to what the organism is

about and the cells communicate both in the embryo and in the adult organism by

sending messages around that they have very elaborate mechanisms to keep most

messages out but many of the diseases that we fall prey to including AIDS have

to do with cells not being able to keep certain messages out so that translated

to me into take the biological cell

metaphor and try and translate it into machine terms it's like you have lots of

separate little computers running on a network able to send messages back and

forth to each other when you think of this as the ground base of

object-oriented systems then it's very difficult to think of this machine

commanding this other machine because real computers have mechanisms for

holding off commands they don't have to listen to you if they don't want and so

the idea that each one of these objects is responsible for its own protection as

the strongest thing you should ever be able to do is to be able to beseech

please do this for me but you shouldn't be able to say do this for me any more

than you should be able to do the equivalent of an assignment statement so

one of the major things I worked on when I was designing the early small talk was

how can we get rid of the assignment statement as an imperative and replace

it with something that was much more of a suggestion and the amount of state that got changed

if any was completely up to the discretion of the underlying object to

me any language that doesn't implement that fully is not an object-oriented

language because you've what you have is something more much more like abstract

data types where you have a bad old idea and you're trying to put Bearnaise sauce

on it in the hopes that will be better but Bearnaise sauce on a hot dog doesn't

help so the input basic two implications

here is that you have an outside and an

inside the outside is completely behavioral messages can't be commands

and assignment has to be replaced by advise if you have those two things then

you have object or whether you have inheritance or not is not nearly as critical not nearly as critical because

I'm a good object-oriented language you can actually build it and you may very well want to in some form of delegation

mechanism so on these mysteries started

becoming clear when I worked on this machine this is the old flex machine

that Idi Cheadle and I did 20 years ago 67 to 69 this is what it looked like on

its own display and it was influenced very much by sketchpad by angle Bart's

first hypertext system by simulant had a

programming language I designed called flex it had a user interface that was so

well designed it most people reacted to

it the way you would if you opened up the lid of a garbage can and saw something moist and fuzzy down at the

bottom or pretty much the way people react to

APL today I'm really proud of that user

interface of all the things I've

designed it produced probably the most startling reaction but didn't have a few

ideas on it it had a really good windowing system one of the first and I

had multiple windows it was a calligraphic display so the windows didn't overlap back then but a lot of

the things that we're familiar with now in modern user interface were developed

on this machine although kind of ass backwards most of the time so after

wandering about what we should do I realized that McLuhan made this great

statement that I read one day said I don't know who discovered water but it

wasn't a fish and I realized that as a

graduate student I was the world's worst user interface designer because the

whole thing about being a graduate student is to learn a really stupid system with 500 obscure commands just to

prove that you can when you turn a person like that loose on a user

interface you wind up with something like ms-dos

so for a while I was thinking well I need some other context there must be

some other context and I looked at some of the language work with chimpanzees

was rather hard to get hold of chimpanzees the Xerox corporate

executives who would never stay around long enough

so finally for various reasons decided to work with children who are also an

alien species but Palo Alto had had bred

hundreds of them just for our experiments and once we started thinking

about children as the end users lots of things changed because children had to

have something so portable they could carry other things too

in other words 0.5 herniation spur block is not portability so about 20 years ago

I came up with this idea which is called the we call the Dynabook back then in it

my friend Dave Liddell who is here today

when he was at the University of Illinois and worked for Owens Corning

worked on the very first flat panel display which I saw a one-inch square

version of back in 1968 and realized it was only going to be a number of years before we could put all of the silicon

that was in the Flex machine on the back of the display and that this would be

one of the inescapable form factors of the computers of the of the future so

once starting to think about that and thinking about the children as the end

users it meant that things had to be much much simpler than we'd ever done

before and one of the hints I got was from a great system done at RAND

Corporation a Rand probably has contributed more to user interface

design than almost any other single place nobody knows that probably nobody

in this room or maybe only one or two people in this room remember a language

called Joss Joss was designed in the very early 60s on a machine called the

johniac and Joss was probably the best

teletype or typewriter oriented programming system ever done it was the

one that invented the interaction scheme that we now know of in APL of having two

in direct commands that came from Jos basic adopted at many years later and so

forth and Jos had the misfortune of being implemented on the wrong machine or else we would have been able to start

much further along than then a language like basic so this language have been

developed for non expert users who are mostly economists who had to do a fair

amount of computation and the these

users loved Jos fact I get a chance to use it in the 60s as well and it was

just there's one of the few systems that you've ever worked on in which there was

a pleasant little tingle when you knew you were going to get a chance to work

on it for a few hours in the afternoon how many systems can you say that about it was just encrypt there was nothing

left to chance everything was done beautifully aesthetically and so forth

and the only complaint these economists had was they said hey none of us can

type we love this system but we can't type can't you do something for us so

the RAM people went actually went through their waste paper baskets to see

what they did right I guess not too many divorces were caused by that these guys used to write

on legal pads and so forth and they made

diagrams and so the RAM people in 1964 invented the first tablet was called Ram

tablet I remember when we got one at Utah a few years later 1967 one of those

Rand tablets thousand by a thousand stylus pointing device first good one

ever done 1967 that single device cost

$18,000 and you had to hook it to a machine that was worth a half a million

to a million to make any use of it a year after 1965 a guy by the name of

Gabe groaner did the first really good gesture recognizer and a few years later they

put this all together into a system called Grail for graphical input

language let's run the second cut on tape one here's what this looked like 20

years ago

this list takes us back where we may

start to edit the flow diagram the no keyboard first we erase a flow arrow

then move the connector toddled away so that we may draw a box in its place

it recognizes he wants a box and just makes one the printed nine used as

commentary only in this case the box is

large so we make as where modern-day

window control came from literally Android flow from the connector to the

box attached a decision element to the

box and joy flow from it to scan we then

erased the floor arrows attached to the process post new area and move the box

to a new position this allows us to draw

a new box then chop off its corner and

label it sub scan with a residual error

I notice it misses the end here he sees it he has to thing but it's motiveless

so just continues what he was doing it makes the change correct the label add a

decision on sub scan so that control may flow to the pleat the diagram from scan

to post a chance for an hour really

changed used interaction or synching your hands right display the structures

underneath so it wasn't just personal

wasn't just going from the mainframe to the desktop this was intimate because

you were right there there were fewer barriers notice that the command scheme

is not even menu-driven here the commands are actually analogues to the

results you want to have so it's not just icon it's an illogical very powerful set of

ideas again done on a machine almost the

size of this this room but slightly less powerful than a 512 K Mac but this is a

technology that actually got buried we we did many versions of this kind of

stuff at Parc I used to show this movie every couple of months to my group so we

wouldn't forget what it was that we were trying to do and the we're ready now in

the next few years to see a rise of gesture recognition because the micros

are now powerful enough to do this and still have some cycles left over for

doing work for the for the user and you

may not know it or not but actually one of the best groups in the country in doing a handwriting recognition is at

IBM big surprise but you actually have

one of the better technologies here and

so if you're you're interested in this I suggest you look in the IBM phone

dictionary for the group that is doing it now John Sculley was once wearing how

many scientists that were at IBM and I said yeah John there are many more

scientists at IBM there are employees at Apple but don't worry IBM doesn't listen

to them just like it Xerox you know you

want them to let you alone when you're working but then you want them to come around when you've got something

[Laughter]

it's hard to get both to happen so I

want to intertwine some stuff in here because in although this isn't really a

historical thing it's very to me it's very important that the stuff that gave

rise to what this symposium is all about was not at all disconnected from the

fact that there were going to be users working on the on a system and so the

the whole thing that brought forth real object-oriented programming and small

talk and the user interface that went along with it was that we had these children and we wanted these children to

not just be able to have access to things but we wanted the children to be

able to build things so our if this what we said was is if

this is if this were print literacy we don't want to teach the kids just to

read print literacy isn't just reading it's also writing we want the kids to

somehow be able to write and the equivalent of writing on the computer is

not necessarily programming in the in the COBOL sense but it certainly is

building structures of your own and so very early we got into this whole area

of wondering well how does the mind work

if at all and the first thing we

discovered that whatever common sense notions we had about the mind working

they certainly weren't at all in accord

with what had been discovered in the last couple of decades

for instance common sense idea that's been in history for thousands of years

is that because we think of ourselves as an individual we think there's a single

eyeness and even in the work of Piaget we see in thinking of the progression of

an infant mind to an adult mind as something like the transformation of a

caterpillar into a butterfly that you go from something less form to something

more form but it's all made out of the same stuff whereas the the truth seems

to be quite different and quite startling that our mind is really made up of a kind of a patchwork quilt

modules or objects if you will and these

objects indicate imperfectly because they came about at different biological times for

different evolutionary reasons and it give you an idea of the modularity of

the mind let's take a look at this and let's see who in the audience can spot

what's wrong with this picture besides the fact you've all seen ups and

upside down faces before who can see

smile is what upside down and what else

these teeth are and what else the eyes okay so now if we were in school and I

were a teacher I would say to you this is the way things are taught in school

by language I would say that what we've

done here is we've taken a picture of a young girl extracted the eyes and mouth

turned them upside down put them back in with an airbrush and then turned the

entire picture upside down okay so having said that you should be perfectly

prepared for what it looks like right side up

now I'm gonna turn this back as I've discovered discovered nobody will listen

to me when it's on the other way so

that's interesting that I gave you the exactly the same amount of information

two times one through language and one visually and you had a completely

different reaction to the exact same information that's the kind of thing

that perceptual cycle and cognitive psychologists really love because means

that different information going into different channels is going to different

parts of the brain and you're having different reactions therefore there are modules in this particular one the

module that's causing this effect the two modules are actually known

physiologically they're located on the under right hand side of the brain where

you do face recognition and about an inch apart from each other one of them

is a expert in recognizing face like things as the thing allows you to see

faces in the clouds when children get scale scared at twilight with menacing

shadows and so forth that module is going off saying I see a face like thing

right now it's reporting I see an upside down face like thing about an inch away

is another module that's only interested in eyes and mouth it doesn't care where they are and it's

reporting that's a good looking mouth and those are okay eyes and so the two

descriptions basically have very little interaction they're quite independent

reporting's from two different modules now when I go to this version the face

guy is saying that's a right-side up face like thing and the eyes and mouth

guy is saying something is very wrong here

now you can imagine why nature might want you to be startled see a face like

this up close to might mean you're going to die on the next 10 seconds and in

fact it's this is not something that you learn it's actually built into our

nervous systems from birth into most mammals in fact so if you try this

experiment out on a dog I suggest you pick a small one

[Applause] so important idea here is that there are

modules the just what the modules are is

still open to debate some of the modules

are even metaphoric I'm not all of them are physiologically cited because of the

way the the nervous system is organized they can act we can have modules that

are distributed all over the the brains of going ideas like left brain and right

brain are a particularly weak way of trying to characterize it but most

important thing is once you realize that there's more than one module and that

the modules have different ways of thinking about things and are somewhat

weak recalcitrant and unable to talk with each other then you have the basis

for a much stronger user interface design or curriculum now the ideas that

led to this user interface that norm was talking about it was developed at the

same time as small talk came from a Harvard psychologist by the name of

Jerome Bruner and Bruner had redone a

number of the experiments of Piaget but

with some twists so for instance Bruner took a child he was in this middle

Piaget stage the image stage and is

where you take a squat glass of water and you pour it into a tall thin glass and the child says yep this was more

water in the tall thin glass and what Brenner thought to do is to then occlude

the tall thin glass with a piece of cardboard and ask the kid again and the kid said oh wait a minute there must be

the same amount of water because where would it go and Brendan would take it

away and the kids would say no look there's more there's more Brenda would cover it up and they could say well hey

wait a minute so if you have any six-year-olds you'd like to tour man

and burner did booty did many experiments in the 50s and early 60s and

the conclusion was that what we didn't have was a progression of stages of the

caterpillar going to the butterfly but instead what we had were multiple mentalities and a change in dominance

because brunner could show the existence of the previous and mentalities to come

by various tricks and Brunner said of

well there are lots of different mentalities and he picked three that he

could show the existence of and concentrated his curriculum ideas on

those and those three were a kinesthetic mentality identical with the PJs first

stage where a whole is to diggit an object is to grab it the iconic

mentality that we just talked about and then around 11 or 12 a symbolic

mentality where facts and logic are used or at least in Swiss French children

around 11 and 12 American children have

a very difficult time ever getting to the facts and logic stage because we

have an enormous iconic barrier between them which is called television now the

existence of these three mentalities they each they each have striking

characteristics and in a nutshell as I

can only go into here the kinesthetic mentality is the one that places you in

space not your visual mentality as you

might think because you can blind a person person can be blinded and still

have a perfect sense of where they are in space but if you remove the for

instance a person's limb from their body map as sometimes happens in an accident

then they may have a perfectly healthy leg attached to them but they cannot

feel that it is there and they they relate waking up in the middle of the

night with somebody else's leg in bed with them and in fact they have a very hard time looking at their leg to see

that it's actually attached to their body there's kind of a blur there

because of course we see not what our eye sees but what our brain knows we see

a simulation that is pieced together and our brain has had its knowledge removed

that this limb is part of us and is very hard to actually what it what it actually yes so the

kinesthetic sense is the one that places us in space it is the one that when we

when removed is the most alienating contact one is the deepest one the

oldest evolutionary one the tactile

sense the image one has two main characteristics that are worth noting

one is the ability to remember images after only a brief exposure is

astounding hopper who is a psychologist in the 60s

in several experiments where he showed 2,500 photographs spaced 10 seconds of

parts just a bunch of slide trays for hours until I seen 2,500 images all

sorts of random pictures and he discovered after 30 days that all of the

experimental subjects had a better than 90% recall of those random slides and

could even identify occluded parts on slides and so forth this is an

experiment you can do yourself get a friend to manipulate a cable TV movie

channel and have them dial into a movie

you haven't seen for 18 years and see how many frames of the movie you have to

see before you can identify it most people can do it in under 90 frames

three seconds of exposure is quite

enough for them to say yep I've seen this movie before and I know it's gonna here's what's gonna happen next I think

about that means 18 years ago you saw this movie you're being put into the

center of it without any preparation and stuff and for some reason your visual system can organize all of those pixels

or what the pixels have turned into to be able to perform that recognition so

it's striking recognition is also fast

so you put a hundred pictures on a bulletin board you can find a specified

picture four times as fast as you can find the word in a list of four on the

list of 100 words so it's much more efficient and this recognition in

comparison and stuff is done by a completely different part of the brain

visual system is also the mentality that is least blocked because it's very

modus operandi is to flit around and take in different parts not to become

fixated in one thing in the visual scene so it's there are many things you can do

that in which an access to visual scene

augments the short-term memory and boosts creativity then finally the the

facts and logic system which is what we concentrate on in school unfortunately

in its strong in in many ways because of

its abstractions but it's actually the weakest mechanism that we have it only

works when there's a strong context so when we're thinking about these ideas

but we had this basis from Brenner and it took us three or four years of

experimentation to come up with this

model that what we what we really should

do this is something has been misunderstood about this user interface because it's always talked of as the

graphical user interface but in fact it was an attempt to combine three

mentalities into some kind of synergy so

the doing with images make symbols says there's some kind of progression which

Brenner identified as well that if you learn where you are learn things tact

alee that gives you a much better set of

hooks for your image system to put things on and then at the very end you

have something to abstract that you want to princess if you want to learn physics

the best way to learn it is body physics and visual physics before you start

looking at calculus then you have intuition and Bruner claimed that and I

believed him that what we call intuition is simply these mentalities that don't

talk sort of vaguely whispering to us

somewhat variant thoughts than the than the one that the one we call eye

actually does so if you take those ideas you the way we picked and we didn't even

like the mouse but as I said tablets were expensive and so we took angle

Bart's Mouse and made a better one and let that be our pointing device if you

believe what I'm saying about the user interface then would suggest number one that we should

have better pointing devices and number two that the pointing devices should

start to give us some force feedback back into our hand because then the

kinesthetic contact would be much greater and the windows and icons are

for the purposes I've just mentioned and we came up with small talk to be the

underlying symbolic framework for all of this stuff this is why the on the

Macintosh and the Macintosh imitators that the pointing scheme is not

functional you don't give the command first and then be stuck in a mode and go

out and then try and pick parameters for it what you do is you do it exactly the

same order that the you do object-oriented programming and that is

you designate the objects of interest first and then you send the messages so

translate it to user interface it says selection first messages later and that

gives you an approach to doing modeless interaction and here's an interesting

idea that we implemented at Parc but in

fact has never made its I just cannot believe that this idea has never made

its way out into the out of the world because it is a direct implication of

object-oriented design the implication of object-oriented design is not cutting

and pasting that is absolutely at odds

with an object-oriented user interface the this is the implication of

object-oriented design this is a document that has a variety of different

paragraph types in it one of them holds texts and other one holds pictures and

these are as norm was pointing out these are the applications themselves so you

don't go out to another app you don't go out in this system you didn't go out to

a painting system to make this picture the painting was actually developed right in the context of the document

because you just started a new paragraph type there and what you see here is a

this is a page from the Lord of the Rings and the cursor has just gone into

this picture and what has happened is it's sprouted a menu because it thinks

you might want to edit it further so as you go into each each paragraph type it activates the sub

activation if you the sub if you will so

that all of the things that you're working on are in the context of your

result it's the thing we call results mode interaction which i think is one of

the most important aspects of WYSIWYG that hasn't really made it into the

outside world always keep the user working in the context of their results

that means they shouldn't go out other applications to get values and bringing

them back they should be able to develop those the values that they want from

within the final thing that they're working on

now this next in order to talk about

this next next part of the talk a little

more context if we think about the two ways that humans have extended

themselves over the last several hundred thousand years one of them is by making

amplifying tools physical tools like the lever and the wheel figurative tools

like mathematics like language itself and so forth all these I think of as

being extensions of manipulation but even mathematics which we think is

something fairly abstract is actually a

way of taking the hopelessly abstract making it into concrete things that we

can then move around so all of these things I think of as being manipulation

the N word is manipulation and the other one and of course the Macintosh user

interface is that style it's basically make everything into an

object and was norm mentioned if you take things that are verbs in English

and make them into objects you have a much better time of it because of course the universe doesn't seem to have any

verbs right English has invented to talk

about horses and cows and plowed fields and so forth and it didn't anticipate

modern physics which as far as we can tell there are no abstract verbs out in

the universe instead what we have are things that are much more like objects

in process so when I hear a sentence like the clock runs down

or the boy runs down the only way I can come up for any meaning of what runs

down actually is all about is by thinking of it as an attribute of the

subject of the sentence okay and there

is enough metaphorical similarity between the two items so that we can use the same word but I always think of what

we think of as a verb as really an attribute of some subject otherwise the

sentence wouldn't make any any real sense and that is an accord with with

modern physics so the other way we've

extended ourselves over the years what I call goal cloning goal cloning means

that we simply are able and willing to go out and convince other people to take

on our own goal structures that's how we get most things done so what's going on

here we have a group that has for the moment has fewer goals than the sum of

its parts and companies are that way

countries cultures and so forth sometimes the methods accrued as in the

way the Egyptians got the slaves to build the pyramids sometimes they're slicker like in the

modern television advertisements mumford called this building a mega machine he

pointed out that over most of recorded history most machines built by human

beings had human beings as parts and the M word for this to me is management that

you manage that kind of process and we manipulate objects and we're going to

see a transformation from actions having to do with objects into agents which we

are going to manage now let me give you the examples the first is sort of an

example of what you can do with a tool

and what a tool really means so let's show the I want to show tape number two

this 2012 young girl who never lived

world not insulated by Akash her mother's my aunt both are there and

father work at home if another father as an act to her this is an acknowledge

that one around your warrant we do think and some paper technology is what Israel

never see anything I fast use it yes in computers I Karen told she injected and

I don't mean you sir today my did one with her anis governing athlete for here

has been stolen disks even though the

belt is you inter favorite children I never even it is about child anywhere here than I was quite intrigued yeah

yeah here says doing visible you man acne and of course you don't have to get

them out I thought well I'll go along

this is arrested but wasn't me impress until I saw what havin a she wants a fresh sheet paper so she used a closed

box in the window and user Ultron's the pop up and goes to the plan F rating

okay implications here it's kind of

interesting it do the user interface at all an accord with what's actually going

on inside of people's head then you get something that can actually extend all

the way down into childhood and be to them as fundamental as the marking

instruments that we grew up with and I think this is very important because I

believe that the the children that can form themselves by being around this

media as media are going to be thinking

and doing things very differently than the way we do this is another example of

in the long line of how we act smarter

by changing our representations we go from Roman numerals to Arabic we go from

Ptolemaic the Copernican every time we change the representation system our

weak systems of logic are made that much

stronger and I think of user interface main aim is not to present but to

represent and it's figuring out what those representation schemes are going

to mean in terms of leverage they give the user this is the most important point that point has been missed in most

people who are doing user interfaces today including a lot of the people at

Apple and certainly the the Apple imitators because they think that the

whole job of the user interface is to present old-style applications with

multiple windows and pull-downs and so forth nothing can be further from the truth

most important aspect of the multiple windows stuff is not to have different

applications up there because as I pointed out that's not the way you want to go after different parts of the

things the most important thing for having multiple windows is to be able to

present different points of view on the same thing marvin minsky says you don't

understand something unless you understand it in more than one way and

almost no one who uses these user interfaces thinks to sit down even

though Apple showed them in the finder that you want to have multiple views of different kinds of things thinks of what

the leverage means is to be able to look at the same kind of thing for multiple points of view and what it means

pedagogically is once you've seen that you realize that if I only know one

point of view about something I don't know it in fact I'm likely to be misled by it so now let's take a look at the

other way that's just starting to come

in and think of why it's going to why are we going to have management of

semi-intelligent computer processes now

I think it's because we're gonna we're starting to be networked up and there

are a whole bunch of things that networks give you as opportunities that

we simply don't have the time and the scope to deal with give you an example

when I was a kid I used to get questions answered by going into the into a

library and once I was visiting some friends in Washington DC and I had a question so I demanded to be taken to

the library I'd heard there's a library in the town and we got to the Library of

Congress and I discovered that the card catalog which they didn't even allow you

to go in was much larger than my home town library and instead of

having you do the browsing they had people who would meet you at the door

their job was to absorb what your goals were and then apply their expertise of

the facilities to that so they became your agent and they even would energize

other agents so my my problem turned out

to be fairly interesting for one brought in by a nine-year-old and so he had five

or six adults scurrying around trying to find the answer to it and it left a

lasting impression and this notion of

agents is as old as computer interfaces itself it goes back to McCarthy's

original advice taker paper it basically says that when you have an information

utility there's going to be an embarrassment of riches organized in a

way so that no human will have the time to go after the stuff no human will be

able to remember all the things they might be interested in so something else has to go and do it now the problem with

agents in the user interface sense is that AI has not progressed much since

1970 it was extremely difficult to build an agent I have built several over the

last eight years or so just to see what they would would be like and they're

excruciating to build one of them was a system called News peak at MIT that

stays up all night trying to find for you the newspaper you'd most like to

read at breakfast so it what it does is

to log in to a dozen or so information

sources New York Times Associated Press Nexus Reuters looking for articles it

thinks you might be interested in it reads the articles to the best of its

ability if it sees a famous name like Mitterrand it has a video disc of 45,000

pictures of famous people it's like a newspaper more get picks a picture of

MIT Iran did have there's a map disk with 37,000 maps and so forth uses an

expert system to format the page and you wind up the next morning with something

that is very much like USA Today

and in often with about the same amount of content

but there are some differences because the headline might say more fighting in

Afghanistan or it might just as well say your three o'clock meeting is cancelled

today because one of the 12 news sources it logs into is your own electronic mail

it reads it while you sleep and if it spots to anything that looks important

it puts that in the newspaper the next morning a sidebar might say your

children slept well last night because remember what we call news is entirely

dependent on the communications system that we have in the late 1700s the news

was all about depth the Federalist Papers were published in the newspapers

of New York City something would be unthinkable today actual essays but

people had time to read them when the telegraph got invented all that went

away because they couldn't possibly have you reading anything long that had to be

thought about and understood because you'd still be working on it when the next news came along so currency and as

McLuhan predicted as soon as you start hooking up news to electronic media you

start getting fragmentation so that's

the idea of an agent that newspaper is something that people would like to have

but nobody is going to sit up doing it even with the best sequel type database and so forth

because it's very hard to even remember all the things that you're interested in so the agents also are going to be

somewhat anthropomorphic this is a real problem because they aren't going to be

very smart when we start off we have to in the user interface design somehow

subliminal ii project exactly how smart

they are and still make the user now we

want something like a puppy dog that goes out and gets your newspaper most of

the time it only occasionally makes a mess on the floor so let's let's take a

look at this is a video we made for John

Scully and we start off at the 1988 Winter Olympics in HyperCard HyperCard

stack that ABC television used to keep track of things and then we project four

years into the future and see what things might look like in 1992 with the

user interface that is still mostly manipulative but has a little bit of the

of the management aspect to it let's roll tape number three please so much

this Bob's math Calgary if you click on

any Olympic site you see a list of the events scheduled for that site you can

get more information about any event by clicking on the name let's say ski

jumping background maybe you can brag to

the last or the main map and then

continue exploring here at the University of Calgary I see the figure

skating events are running now that's a sport my family's interested in so I'd

like to learn more about it the previous Winter Olympics took place at Sarajevo

in 1984 when Macintosh was barely a month old

imagine what Macintosh together with hyper card will allow us to do in

another four years by the time of the next Olympics

[Music] we will require powerful techniques to

help us find and manipulate volumes of images text music video and applications

we might by that time see the beginnings of intelligent agents or perhaps by then

they'll be only semi-intelligent Phil

today is March 3rd 1992 you have several data service articles ready and you have

ma'am this is my agent I call him Phil Phil

open the 1992 Olympic stack

the 1992 Winter Games are being held in Alba V friends let's get a closer look

at the town let's look at ski jumping

again we can play the video

[Music] we could learn more about this sport or

go back to the math let's look at figure skating

I'd like to see who's on the USA team and I'd like to see Debbie Thomas I'm

looking for a spin

like that let's go back alan kay was

explaining why skaters go faster as they tuck their arms in he showed me a

simulation to illustrate the underlying principles let me call that up Phil add

the physics draw toolbox which tool box

physics draw let's see I'll need a

counter we can make the window

translucent let's step through the spin

and count the frames okay that's one

complete rotation now I'll draw a model

representing a skater with her arms extended select the center axis and

rotate how fast to rotate this speed and

here's an overhead view of the figure let's hold on to that earlier Allan and

I created a similar model where the skater has tucked her arms in and

appears to be spinning faster Phil display skater a model 1 let's compare

these two models

the small one the one with the shorter radius seems to run faster but let's

check that I'll set them both to run for three tenths of a second a small one

turns farther around the circle but let's see how much distance is actually

covered I'll straighten out the curved paths to measure them both paths are the

same length even though spin increases the distance

traveled is the same the luge event is about to begin show it live or record it

no I'm not interested in luge I don't understand I like skating

I don't understand don't show luge don't

record it I understand

I'd like to say this simulation Phil start a link Phil show the stack map

too far okay show recent link to this

background now anytime I'm on a skating

card I can link to my physics simulation by the time the Winter Olympics come to

a birdie in 1992 CD media will be well

established each of us will be able to choose our own path through Worlds of

entertainment and information and tools like those that we've been seeing will

allow us to integrate the full range of sound video and text information this

demo ends here John should I show the

pre-recorded speech or will you give it live humane things to think about it

here are that although we are of course

going to go to hypermedia and it's going to be object-oriented it's really not

going to be complete until the user is able to do active thought experiments

experiments most hypermedia up till now is assumes that the user is going to

navigate around stuff that may be active

but the user isn't going to be able to do a lot of creation themselves this is

a critical thing to to realize and that

the agents of the future are going to be helpful mainly for their ability to work away from us not from the amount of

comprehensiveness that they have the best object we have for exploring this

new area is the one between our ears providing its active thank you

tell some questions yes yes

well you can do that but just consider

the problem of a system file that some people have access to and other people don't have access to all right there's a

perfect example of when somebody's what you really like to have is when somebody sends a message not a command but a

message to that system file you'd like to have some way number one of checking

the access privileges before you do anything and number two and you'd like

to be able to do that checking quickly okay but even being able to do the first

would be good whereas in the current things that are called object oriented

languages you are already committed you've already formed an activation record because it's a procedure call you

are already inside the method and thus each method that is sensitive has to

have the responsibility access you can

put that in you can put all through a procedure and you have to remember the next time you do that it has this

problem that isn't all I mean languages

also allow your things that allow you access this way I'll talk allows you to

turn a thing called a block that block is a particular kind of object that it's

a block context actually that by virtue

of what it is it has access to some of the internal variables of the object and

so it's you can create Trojan horses and

so forth and but basically what happened is when they were implementing the newer

versions of small talk some of the older people who worked on it didn't work on the newer ones number one and number two

most of the newer people had never done any operating system type stuff and so

the operating system type stuff that was in the original one didn't get put in as

the later the later small talk went back to be looking a lot more like similar

than the original one did the original one when you sent a message you weren't

calling a procedure at all all you were doing is knocking on it and saying hey

you know it's the equivalent of the postman leaving a thing that says there's a message waiting for you down

at the post office if you would care to get it it was that arms off relationship that

small talk 72 had that allows you to implement all the operating system

protection ideas that are important and I think that the the ability to

implement large systems in object-oriented languages as well as

being a being able to implement them conveniently is going to have a lot to

do about how much protection integrity there actually is it's easy to do you

just have to know that it's important and put it in the in the system yes

right

yeah well I think again I had a there's

a little technical part of the talk that I left out that but I think that one of

the things that's going to happen is that the way we're thinking about objects now just even as people are

struggling to use them one of the when

you look at object based systems today they're mainly modular in terms of state

right the state is sealed off and sometimes it's not even state because

you can have procedural interpretations for things and the thing that's not

modular in object-oriented systems is control in fact sometimes gets worse

because you can have what's called a buck passing theory of control where you

constantly pass and control all over the place and eventually it gets to somebody

you can handle it and they handle it and it's both through inheritance and also

through containment that that happens when you when you start looking in the

system that's been implemented that way you start realizing whoa all of a sudden you have to start understanding things

again and the whole idea of object-oriented programming is be able to program things without having to

understand how everything works it's like Prolog it's like magic for the

Baths the first three hours and then all of a sudden you have to understand ten times more than you ever did so it's

it's kind of a cruel trick and if you

look into biological systems again the first small talk was designed

more this way it was less procedural in

many ways and it was hoping to use a control structure invented by Dave

Fisher who did a great thesis at Carnegie in 1970 called the control

definition language that was one of the that was one of the inputs to the design

of small talk and there he was very interested in what you might call a

gorrik computing and the the current

stuff I've been fooling around with our systems that don't send messages but only receive in other words one of the

ways of getting around the control problem is to if you're a module what

you do is let the system know what you're interested in it's up to the system to somehow get that in for

to you you don't care how what you don't want to have is a system which every

time a new object goes into it some other object has to know it should send

it a message to write because then you're right back to something analogous

to the data structure problem having random procedures understand the formats

of data structures and here we're talking about having objects have to

haven't understand essentially control formats so I've been working on a design

of a system that only receives but it

receives by indicating things that's interesting interested in and then the the system constructs a structure that

will guarantee to deliver that information to it in a timely fashion and again I think if you just get away

from software systems and think about networks again it will tell you a lot

about what to do with objects because you really you can't take objects

seriously unless you think of a transmission line between them it's too easy to just think of sending a

procedure call to them and you don't have any of the interesting things that have happened with concurrency and

synchronization and protection and all of those things that are really going to

be a major part of just standard

programs that we're doing in the world so I think you just think of it as being a whole bunch of net networked computers

crying to cooperate and realize and what the problems there and how you want to

write the code then I think it tells you what to do with objects that was the way

the original small talk was designed anyway yes

right good question the I don't actually

have any of those things in mind but when when Smalltalk 72 was done 16 years

ago the idea back then was what

capabilities could do if abilities are a

short way of dealing with access rights

okay and the access rights there our instance are at the instance level in

capability systems and I wouldn't think that trying to protect at the class

level is C it's easy to protect the class if you can protect at the instance

right because the class is an instance of class class that's trivial to do it

so you want to protect at the instance level its itself and protecting at the

instance level in capability systems is tantamount to protecting each message

individually because that's what a what a capability map actually does for you

so it's a very nice way of doing it yeah

sure no I don't see any if you can deliver it

for that price I think he will sell a few but as far as I can see there's no

in fact the interesting thing is the user interface a lot of the things in

user interface are more like small talk than than than the Mac he has

hierarchical browsers in it and the proportional scroll bars and other kinds

of things that small talk and although

he has a very strange scroll nose we saw

at window resizing method apparently I haven't actually used the machine yeah

but somebody told me you can only resize one dimension of the window at a time and he would would have known that Apple

didn't own that if he'd seen the the Grail movie because

well again in the longer in a longer talk what when I started thinking about

it I realized I didn't have not only didn't have any good ideas about AI that

I'd never had I had had a good idea in AI and I figured that was either because

I was completely stupid or else I was thinking about it in a way that wasn't

terribly fruitful or the stuff that

people have been doing in AI didn't have any good extrapolation into what would

be powerful I still don't know which of those three is the is the case but I

have a couple of degrees in biology so I started thinking about it from biological stand the first thing that I

realized was that trying to essentially do a Beethoven which has been one of the

main aims of AI who knows to do some superhuman

intelligence right off the bath to impress the Pentagon sponsors is a real

loser because we can't even do a decent ant and so I got basically I got

interested in thinking about what would it be like if you forgot about forget about Beethoven for a while what would

it be like to just try and understand a I as an architectural problem which I I

do this is I think of life as an architecture right the material moves

through us we recycle the atoms in our bodies every seven years or so but the pattern remains some kind of a weird

feeling to think of matter passing through this force field or so that

organizes it and the same same thing is

likely to be true for intelligence it's it's an architecture and when you're doing doing architecture you have to

start building things and seeing which cathedrals fall down and you're really

in bad shape in architecture if you build dog houses because you can build a doghouse out of almost anything and it

won't fall down and doesn't scale you know the big problem with our whole field is big good ideas don't often

scale and we always get screwed when somebody has a good idea that works in

the small and then they try and scale it off by a factor of thousand and all of a

sudden anymore so the to me it seemed like the

an interesting thing to do would be to try and build some biologically relevant

models of animal intelligence and pick

at a bunch of different levels so that you wouldn't get into the scaling bug so

that you actually thought you were developing some something that might

give you a few hints as to what the what a good architecture for intelligence is

I joined up forces with Marvin Minsky at MIT set up a project called the vivarium

which it involves children because the again it's it's hard to design for

adults it gets boring after a while so the project for children wound up being

saying what would it be like for children to actually build these animals

what would it be like for a child to sit down and build up a both a physical

animal and a mental animal that you could put into a big environment if we have one more second I can show you a

little clip of some of the work that we've been doing this is will be taped

six and what it is is before you show it

what it is is some real stuff and some fake stuff the real stuff is all of the

3d graphics that are in it which we did in real time on a an Evans & Sutherland

flight simulator this is the machine that has something like 3.5 gigaflops of

computation it's about six to eight Cray

XMP 48 worth of stuff so it can actually

generate on a database that has some sub planetary in size

you know because you have to have to fly over hundreds of miles of it at Mach 3

and it's able to perfect anti rastering

perfect texture mapping on the thing and it has it has the ability to have as

many as thirty two independent cameras at once so you can see look into the

same database from multiple points of view so all the stuff that you see in in

this user interface at that level is real and then what we did is a little

mock-up to see what the user interface some of the elements of the user

interface might be for children we're using elementary school children so

they're both visual and very tactile so we got very interested in what would a

tactile kind of user interface be particularly one without any menus in it

so when you so I think you'll be able to tell the difference between the two so

let's roll that tape number six so here

just turn out the lights so here we have actually four desktops in operation

under projects and we go into the vivarium one and here's the flight

simulator we have two different points of view two camera controls as you see

there so this is 60 frames a second

real-time computer we wanted to do it all in real time to keep honest about it

and one nice thing about real-time graphics is you want three minutes of

real-time graphics you it only takes

three minutes to generate so here's the

little kids notebook going rockfish you

see the real real crab over there and the simulated crab and the crab model

that the kids working on here's the sea

snail weird color it has is actually the

color that's how we do scrolling

and here's an avoidance reaction this is what a real one looks like really an

amazing amazing color and down in the

sales simulation here it's just chucking along on its basic movement cycle these

these snails these particular ones have only about 2,000 neurons so here we open

it up a little bit kids love that

various brain models there's three or four different points of view here's the

one that they would use to program it with all the sensors and now if we put

them all to work get to see what the whole environment actually looks like

here

[Music]

[Music]

[Music]

[Music]

[Music]

[Music]

[Music]

this about two years into a six-year project where we basically built all

this stuff by bite simulator brains of

the animals we built with a bunch of thesis projects at MIT that Marvin

Minsky and I are supervising the next phase of the thing is to do a system now

that children can actually build the the animals and put the brains and so we're

now the the idea is about two years from now that is to start testing that aspect

of it by the end of the six years is to exhibit sort of an animal building kit

with brain models that ethologists would not turn up their noses at thank you

very much [Applause]

[Music]

[Applause]

we're gonna take a break and it'll be

ten minutes please return in ten minutes thanks

[Music]