>> So, I'm involved in a project called the Immersive Tools Project,
and I started working on this because there's a program at Berkeley,
which is in the Institute for Arts Education and special needs.
And that institute grew out of a long standing program that
was at the Boston Conservatory, which was connected to
their graduate program in music education and autism,
and then a couple of years ago the Boston Conservatory and
Berkeley College of Music merged, and so, one of the things,
and I've been at Berkeley for 17 years now.
And one of the things that we were very fortunate to have become part of Berkeley,
is that program and that concentration in autism.
So, when I was looking at projects to think about what could I do
to help people with virtual reality, extended reality, augmented reality,
what are some of the kinds of projects in sort of excess of games,
serious games, what are the kinds of applications that
could be of help for people in the world.
One of the areas I was really excited about was
this project and to work with some of the people in that graduate program.
And I have a personal angle involved in this as well,
because one of my children is on the autism spectrum and he also takes
music lessons in the program that these tools are addressed to,
although, he started taking lessons in the program
after I become involved in this project.
So, the program is on Saturday mornings, and there were students not necessarily college
students, but students from the Greater Boston community,
who come from pretty young, some of them are sort of four or five years
old, and there's some group like music-making,
classes and then they go all the way up to adults,
and so it's a chance for some of the graduates students to work with.
People have students for private lessons, and then also some of the graduates,
and then some of the faculty in that program as well, in that department.
So, all around it's been an amazing and very quick growing program,
where there have been many, many benefits beyond what people had originally
hoped of course as it often happens with special needs and arts,
where there are the possibilities for the specific population to have teachers and
an environment that can best help them blossom in terms of any talent or
interests that they have in that particular art,
in this case, different music instruments and now it's expanding because of just
widening out with the larger umbrella of this institute for arts education and special
needs.
It's widening out in a couple of directions, one to include dance and movement,
and then also to include other needs in addition to autism.
So, anxiety and visual impairment.
So, as this has blossomed out and become a larger more exciting community,
an exciting program, it's also gotten bigger.
Of course, right?
And we have multiple classes, multiple lessons, the conservatory building,
that the lessons and the programs take place in on Saturday morning is really exciting,
it becomes a place where parents are able to meet each other and talk to each other,
and the teachers are able to get to know each other and the children and the adults,
the young adults and some adults become just so much more
confident about their abilities in music, but then of course,
there are other abilities as well, in terms of the students with an autism spectrum.
They often become more communicative with their teachers and
that relationship is so important and becomes a real place of trust.
Well, let's leave it a trust for right now, and then we'll go into some other things as
well.
So, as we were approaching this project, we used the process of design thinking,
which made famous by the Stanford D School where the designers first empathize with,
and think about who is going to be using this thing that I'm making,
whether it's an object, or a game, or a tool, whatever it is, an interface.
I like this picture that's up here actually quite a bit,
because even though it's an older graphic from the D school,
I like the way the arrows go out and then they go in,
and then they go out again, and then they go around.
I think that's a really a nice image that they have.
And first to empathize for designers, and of course this is a really important part
of working on any kind of tool for ASD,
because it's the idea of creating something that
is going to be helpful for and useful for someone who,
if you are a neurotypical designer, for someone who is not understanding,
processing and experiencing the world in the same way that you are,
and most importantly and not in the same way that the whole world is set up for.
So, how did we get a group of people who aren't necessarily thinking about that
to think, well what does that mean,
then to define our tools, then to create lots of different ideas,
where really is still in the prototyping stage now,
and now we're starting to get feedback with bringing the tools to our students,
and also one of the things we realized during this process,
is that the teachers were as much group of users,
as the students that we were hoping that they'll be using them with,
which was sort of an interesting thing, you first start to think "Oh, these are,
the tools for the students the music students," and this is what they need.
This is what we think they need, this is what my research suggests that we
need, and I'll tell you a little bit more
about our team right here.
So, that's me and then Dr. Broder Bernard, who's been working in
music education and autism for a long time now,
and has just done such pioneering work.
She's such a like a generous, and gentle, and incredibly energetic person.
She started that music and autism Saturday morning lesson program by herself,
and believed in it didn't have a lot of resources first,
made it happen, and has watched it become this institute,
and there was a big opening Berkeley style, where we had a wonderful party,
and music, and a reception, and it was a wonderful moment for
her to have all of that kind of be there and be
recognized and blossom into this new level of a program with a lot of
good support, and then we have a student team.
So, we have three graduate students in the music education autism program,
and all three of them are public school teachers in Boston,
or right around Boston.
Music teachers, music educators during the day and then they go to
the graduate program at night, the master's program at night,
and then two of my students who are in the video games scoring minor at
Berkeley in the film scoring department in electronic production and design program,
and so, they're musicians, they're composers, they're sound designers,
and they're really two of my stars, and they've been terrific.
Both of them showed interests in accessibility, one of them had come up
with just different course with me, had chosen to do his project,
like a Unity project for, to try to simulate what it would be like to
be blind.
It was his idea that he did on his own thinking" Oh virtual reality
might be good for that, " and so, he was one
of the people I chose to be on this project.
And then Courtney as well was a very empathetic person
who is really interested in helping other people,
and really interested in working with children as well, excellent with
kids.
So, that's how we gathered our team together, and so I went in thinking that we would have
mostly social stories and knowing some of the difficulties that
my son had experienced with his music lessons that some of the things that if you make a
mistake, that's okay and keep going.
Which is hard for anybody at all, but can be particularly challenging
for this specific population.
Or that when we practice music, that we practice not always from the top,
right, from the very beginning,
but sometimes just a little piece in the middle and we'll just do
that part or will pick up from someplace quite towards the end.
And that can be challenging as well in a way that some people
approach things cognitively from the beginning and going all the
way through to the end every time.
And it's one thing if you're practicing on your own,
but once you're working with other musicians who are
used to working in a different way, that can be more challenging.
And so, those were some of the places I was interested in
where the social and the cognitive meet and can be barriers to where a musician with
ASD can run into some issues when they're trying
to be playing with ensembles in orchestras, in bands, some of those kinds of things which
is, when I think about where
I hope some of these tools can go, that's where I hope that these are the kinds
of skills I hope that will be able to help.
The music teachers were saying things like, let me see if we can get to,
oh, I'm going to just go back.
Music teachers were saying things like, ''Let's work on steady tempo,
let's work on dynamic.''
So, it was really interesting the way that we had and so we've been working on all
of these.
So, I think, so this next one I had to tell the people in the booth that this is a really
loud, has really loud noises and so I would like
you, yeah, please or I could just skip it, what
do you think?
Would you rather just skipped the thing with loud noises or
whatever or does anyone have any strong thoughts?
>> You should do it.
>> Should do it?
Okay.
Okay.
So, this is called too much information.
And has anyone seen any of these kind of videos?
Yeah, so the idea, this is an example of using 360 video.
They say VR and so, it's non-immersive VR and what we have been
working with is what level of immersion do we
want to use in our program.
And there are all kinds of reasons for that.
Some people don't like having a headset, right, put on them,
that that's uncomfortable, right, for various reasons.
Sometimes I don't like having a headset put on me,
but I found these little homemade.
Has anyone ever used any of these?
They're basically the lenses that are in Google Cardboard,
if you're familiar with Google Cardboard.
These are the lenses, so you can just, well,
I mean I know you can't see what I'm seeing but you just go like this and so,
there's nothing on your head and there's nothing blocking,
it's really easy to stop this, right?
And if you have on headphones, and good headphones, then you can have not the same kind of
immersive experience that you would if you put
on like HTC Vive or an Oculus Rift, but it's alright and
it's just so easy and it's easy than to carry these around because that's it.
And so I bring these to classes and hand them out,
collect it back again at the end.
And so, this has been a really good solution for me and all that,
like the Google Cardboard things are, is this in some cardboard?
I got these from- I'm from Boston, so we have Micro Center.
You can get them online, I got them.
I saw that they were on clearance so I bought all of them.
That's how I got my kit, my class kit.
But anyway, so I'm going to play this too much information here and I
think I'm going to play it.
I know, that's right.
Yes.
I'm not having the best luck with it.
Okay, I'm going to, oh, that was it, okay.
Well, I'm going to tell you where you can find it for yourself and describe it in a
few sentences.
It is about a boy who, we see things from
his first-person perspective and he has a soft toy,
an animal that he's holding.
And so you can look with your device and I'll show you one
here.
You can, you can look on your device and, I'm not connected to the Internet.
Okay so, you can look on your device's 360 video and
his mother is having a hard time working on, working to get money out of
the bank machine and he's looking around, he's at the mall,
he's looking around the mall and the noises are getting louder and louder for
him and the colors are getting brighter and
brighter and the lights or fluorescent lights are starting to flicker
and flash and he's really having a sense like
just too much, is too much information,
too much sensory input, and so the noises get bigger and
bigger and it puts you in his first-person perspective of what that's
like.
So, if you're neurotypical and or you don't have
any sensory processing experiences like that or you've never had an anxiety attack,
I mean there are all kinds of reasons why or ways in
which people might have had an experience that's similar to that,
that you would then have something that is like an experience to,
in order to increase empathy.
So, one of the things that we've been trying to think about is,
how can we help people who are neurotypical understand better what
it is like for the student or whomever the person who they know,
their friend, their child understand better what it's like for them?
And there's this idea of autism spectrum and when people hear spectrum,
right, we often think of this line
and I don't know if anybody has seen this from Theodora,
this wonderful kind of comic book style, graphic novels style from presentation
but explanation that she's made of what her experience is about,
what it's like to be, what the autism spectrum is being like and
she says that we think of it as high-functioning and low-
functioning and people are often, this person is high-functioning, low functioning.
But really, the spectrum looks more like this, and this is where I want to pivot more towards
talking about executive function and cognitive accessibility and cognitive issues.
Because as you can see up here, your autism can affect language,
motor skills, perception, executive function and sensory tolerances
but autism spectrum disorders are highly individualistic.
And one person will experience things in different ways,
in different contexts, so someone at home can be
very different than the way they are at school, than the way they are in the hallways at school,
than when they are in a private lesson, et cetera.
So, this is the graphic I really love, because it shows us how someone,
it's not that someone is high-functioning for everything, right?
If they were, then there wouldn't be the traits or the places where they had the disability,
where they had the issues with everyday life.
But in fact, that it says mixed profile, and that there are some traits
that can cause pronounced difficulties in everyday life,
and then, some traits that are very useful, or can be very useful in everyday life.
So, in terms of executive function, we're thinking about executive function
that we define that as a family of mental processes that are
associated with the functions of the prefrontal cortex.
So, specifically, it is the higher level cognitive processes
oriented towards inhibition, working memory, and shifting.
Being able to shift from one thought, one activity to another.
And so, we're talking about reactive inhibition and regulation of goal oriented achievement
behavior.
I make sure I'm covering everything I wanted to cover on here.
So, it is essential, of course, for physical and mental health,
academic achievement, and cognitive, social and psychological development.
So, when we were thinking about these.
Here, inhibition, working memory.
And by inhibition, the inhibition of impulses to do something else, right?
The inhibition of- So, difficulties we thought we could
address with immersive tools.
Whereas, I was talking about earlier, recital unfamiliarity or anxiety and or anxiety.
So, the students, the peculiarity of this situation that the students
in this autism lessons program
can't get into the space where they have their recitals,
before they go to have their recital, because of the way that room is booked on
Saturdays.
The only day that we- it seems like how can that be,
but that just is the way that it is.
So, when I heard that, I thought well, this is something that we can
use technology and media for, to preview.
And we can create VR first-person like the too much information
piece.
But this time, not too much information, like you don't have enough information,
we're going to give you enough information so that you can
preview as many times as you want, or your parents can help you
preview as many times as you want.
And this is another one of these kinds of tools that it would be useful for anybody.
I mean, there are so many issues around like performance and anxiety,
and stage fright for musicians of all kinds.
Having to go and play in different spaces that we've never been in
before.
We're also talking about some of those music specific kinds of things.
We came up with 360-degree video augmented reality
and virtual reality.
And then, we went into this whole idea of well, what's going to work?
I mean, who's going to want to put on the headset?
And then, we moved into a real question about if students are
in their private lessons with their teachers, and here's this connection with
their teachers that we recognize as being one
of the most important aspects about their developing communication skills,
about developing reciprocal relationships, about all kinds of things.
Here they are, they come in the first week, and they're not saying very much,
and then we check in weeks later, and I hear this from
the people who've worked in this program for years.
Check in weeks later and the student is like talking and laughing with their teacher,
and the parents are saying, "Look at my kid.
This is amazing, right?
This is what happens here."
So, we were saying, "Well, how are we going to do that?"
So, that's why we came up with using more augmented reality than virtual reality.
And so, we're moving towards iPads.
I can show you.
Here's the recital preview.
I only have a couple of minutes, so I'll just show you this.
This could be loud too.
So, in the booth, if you can check the sound.
This is not loud.
This is super silent.
So, here.
And then, this is a screenshot of
an Augmented Reality App about Dynamics.
So, the louder somebody plays the bigger that ball gets.
The idea is to get into a certain zone, and when it's not in the right zone,
it's a different color.
So, we've been working on that as well.
So, there's been research and this is only one of the studies that I've been looking
at, that talk about the relationship between
video games and cognitive skills, in general, in cognitive issues.
But in particular, around executive functioning because that's one of
the important ones for us with this autism project.
And this one, correlation between video game mechanics and
executive functions through EEG analysis.
In particular, focused on how specific game mechanics can
develop specific cognitive skills.
This chart, I'll post this on Twitter @loirl, so you can take a better look at it.
It talks about this specific game mechanics that can be used to enhance certain cognitive
processes.
So, this is a good, if you're looking to design,
if you're looking to develop for specific cognitive processes.
And so, I made a chart out of somebody else's research with EEG,
which is not the kind of thing I do at all.
But that resource, that study has all the explanations of what accurate
action, timely action, mimic sequence,
pattern learning, and logical puzzles are, and then which cognitive skills they were
better at, doing the pattern learning and
logical puzzles were particularly good for attention,
and then, other ones are better for memory.
So, that's like the shorthand version of if you want to come up with things
for increasing or working on cognitive skills, that's it.
And then, let's see if this will play.
Maybe now, there's no sound though.
A little bit of sound in the booth?
Okay.
So, I'm just going to.
Is anyone familiar with Autcraft, with AutismFather?
Anyone know?
Yes.
So, I'm just going to tell you a little bit about
this project.
I know my time is almost up, but to me, what is so hopeful and so
exciting about what we can do right now, with games, with virtual worlds,
I think Minecraft is the big breakout virtual world.
The Minecraft generation has expectations for what they will
find in an online platform that is exceptional, right?
They expect that they can mode it, that they can modify it.
They expect that they can be social in it.
They expect that they can access it easily on every platform.
They expect that they will be able to understand it in a range of modalities.
They expect they will be able to use it whether they can read or not.
They expect that they can use it in all of these different ways.
They expect they can hook it into other things like this cord,
and just use it in all of these ways.
They expect that it's theirs, right?
And that is what a virtual world is.
It's a place that is the people who are in it and who are making
it.
And so, get ready for Ready Player One.
That's coming out in just a few weeks.
And if you think people are talking about virtual reality now,
just wait because people are going to be talking about it.
And this is the time when everybody is going to be, not everybody,
but a lot of people and the culture is going to be focusing on what is VR?
What are virtual worlds?
What are we doing with this?
There's going to be, "It's dangerous.
Everyone's just going to sit in their room and pathologizing it."
And there's some serious concerns, of course, there always are.
But there also are so many amazing benefits.
What's Stuart Duncan did with Autcraft, as he noticed that
his child with autism wanted to play Minecraft, but there were people who are mean on these
servers, and there are people who are
mean on the servers, and in games.
So toxic, right?
So, he decided he was going to make his own server, and he did.
And he put up a little tiny notice saying, "Does anyone want beyond this nice server
for autism?"
That's not what he said, but something like that.
You can watch his TED talk.
Watch the whole thing.
I had a little excerpt but you should just watch and you can read the transcript.
It's fantastic.
And what he said is what the kids did in it was amazing.
There were kids who first, they were misspelling things,
but then they saw people spelling them correctly and they learned.
There were kids who parents were saying, "My kid doesn't talk,
but my kid is talking when he plays or she plays Autcraft."
They were emergent cognitive skills.
And that's the secret, I think, or not the secret,
but that's the key to how to think about accessibility, to think about how do we create accessibility,
so people can become as wonderful as they possibly can.
There's always potential.
There are always ways in which we can, in the title of my GDC talk later,
Dial Down, some of the inputs and the barriers, so that people can have greater access to
themselves.
We can change the environment, we can create filters for games,
we can create augmentation for what is going to be in
our actual world and in our virtual world, so that people can modify their situation,
so there is not too much information.
Thank you very much.
Không có nhận xét nào:
Đăng nhận xét