Design and Technology Blog

 Bridget Riley's Fission

Bridget Riley

British, born 1931, 

Gallery label text, 2007:

The distortion of the black dots at this painting's center creates the optical illusion that the image surface is caving in. In this and other early works, Riley says, she "wanted the space between the picture plane and the spectator to be active." As the title implies, the surface seems to be folding or opening, suggesting that the viewer could be swallowed up by it. While the black dots on the white surface don’t depict something or tell a story, as in a traditional painting, the sense of space this painting creates is not unlike the illusion created by the use of traditional linear perspective.

Bridget Riley - Current

William C. Seitz - The Responsive Eye 

Source

Concept

In this experiment, I am using optical flow and blur to create semi visible yet colorful faces and by using face tracker and ofxCv I tried to create random words each time when the person open his mouth. Mouse open also means grabbing a screenshot. If we consider this as a public experiment, each time we will have words and the ghosts behind those words and it turns into a digital ghost portraits archive as the result of this experiment. 

The main problem I faced was to control only mouth and create interaction. I tried to use FaceOSC to track specific part of the face but I couldn’t use properly. That’s why I kept using two different facial expressions. And I still have problems to position words on mouth. 

Github: https://github.com/mehtapaydin/wordless

Precedents

https://github.com/crecord/faceOSC/blob/master/textureMouth.pde

https://github.com/edap/bubbles

Resources

http://www.v3ga.net/blog2/2009/03/gravity/

https://github.com/armadillu/ofxFboBlur

Final Pape Draft: https://drive.google.com/open?id=0B7Fr4CY_fY2HV0NwUnZ5OFVONFE

This week luckily I discovered Bodymovin add-on for Aftereffects. It basically transforms AE animations into JSON files to be able to use in web projects. So, instead of paper.js, I decided to use after effects animations to use in my projects.

Icons

I created new icons for the demo and narrowed down the whole experience.

After the feedbacks that I have received in class I also create one more animations to give 

Branding

While doing this I also decided to create a visual language for the project as a whole. 

Posters: https://drive.google.com/open?id=0B7Fr4CY_fY2Hend2bC0wWmRPSVE

After some experiments about the device to device communication, based on an example I created this functional and more improved mock-up. At this mock up, when the user sends a card from her device it falls to the ground on the second browser. When two cards match in terms of association, it creates a visual feedback. 

In order to create this visual feedback, I started this week to investigate paper.js to be able to use SVG images comfortably. I have a primitive result but I thin even this result is enough to make another user test.

User Test 2

I did my last test with the functional prototype. I got some user feedbacks and I noticed that it is really hard for people to write generated URL from their phone. That's why I decided to change it into QR code and I keep the code on the screen until two players involve in the experiment.

https://drive.google.com/open?id=0B7Fr4CY_fY2HRzY2Tm1aSkZ4cmMBefore deciding the technical details, first I did a design survey to understand the general approach of the users in terms of learning through diverse media devices and specifically language learning. I asked them their prior experiences about language learning, the difficulties that they faced. At the second part of the survey, I wanted them to answer some Italian word matches based on my learning cards. They all were English speakers and they didn't know any Italian word. Their answers were all correct which showed me the associations that I created are simple enough.

User Test 1

After the survey, I did my first user test with 3 years old Greyson with physical cards. Even though my target user group is the primary school students, the user test with Greyson showed me the association between simple words makes sense for him as well. He gave me the correct answers for each match and as the result of our test he didn't learn a new word but he learned what is bacteria from the combination of bacteria + milk = cheese.

https://github.com/mehtapaydin/OF_Week9

Resources

Particle System with GUI https://masteringof.wordpress.com/examples/particles/

The main logic of using fft https://github.com/jahfer/Audio-visualizer/tree/master/src

Songs

Song 1 - Einstürzende Neubauten - Dingsaller 

song -  Ilhan Ersahin’s Istanbul Session - MCCOY

COMMUNICATION FROM DEVICE TO DEVICE WITH WEB SOCKETS

For the engagement prototype, I decided first to focus on to be able to create a functional, device to device communication. I’ve chosen collaborative drawing as a single engagement and tried to implement it with socket.io and p5.js

Local host 1

The first engagement prototype was between two desktop local browser and wasn’t working between mobile and desktop. 

Local Host 2

Second Iteration with Paper.js and Socket.io

Since paper.js has more control over SVG images I wanted to improve drawing experience with paper.js. 

The rest of the week, I improved my concept by deciding main functionality. As the result of my ongoing researches on language learning, I noticed associations are the important part of learning and I wanted to create a playful scenario around word associations as the case study.

Ajit Narayan's word game to communication in every language is very inspirational for me while making research about language learning. He basically creates a schema based on the association between people, objects, verbs and directions and based on this scheme he offers a new language learning model. His ideas become the structure of my idea to create an association based playful experience to learn vocabulary.

Sketches

Based on the idea I created the first visuals to be able to make a user test.

FINAL PAPER OUTLINE

This week after I receiving some feedbacks, I decided to create my experience based on users’ available mobile devices. I narrowed down my concept of language learning and communication. 

TECHNOLOGY PRECEDENTS

ITERATED CONCEPT EXPLANATION

Technology has been revolutionizing the way we live for a few decades. Especially as the use of internet and mobile technologies has dramatically changed our relationship both with each other and everyday life. Today we witness the new definition of socializing, entertainment, learning, creating identities and getting the information on the internet with the excessively use of different kind of mobile devices, especially by the younger generations. They have been redefining our everyday interactivity with the current digital space and creating their own way of communication on it.

 However, since the improvements in technology are faster the ever, we accept them at an equal rate and adapt our behaviors to them rather than adopt our technologies to us. So, when we consider the younger generations, the acceptance of new interactivity is not generating healthy outcomes every time. So what is interactivity and how helpful is in terms of learning? Barker considers interactivity in learning as “a necessary and fundamental mechanism for knowledge acquisition and the development of both cognitive and physical skills. Current thinking about how learning takes place emphasizes the constructivist approach, which argues that learners must actively “construct” knowledge by drawing it out of experiences that have meaning and importance to them [Dewey 1966]. 

Participants in an activity construct their own knowledge by testing ideas and concepts based on prior knowledge and experience, applying them to a new situation, and integrating the new knowledge with pre-existing intellectual constructs; a process familiar to us from real-world situations. 

As group including Steve and Nelson we were planning to create an image carousel experience. Our plan is to navigate the images by using hand and with the finger gestures we planned to control single images to get closer change size etc. 

My part was carousel navigation and I created a class including 4 images and by using hand movement I am controlling the carousel navigation in x-axis. I used ZenoToPoint class exercise to control the movement and speed. 

https://github.com/mehtapaydin/OF_Week8

Collaborative Learning Through Technology

KEYWORDS SOCIAL LEARNING - COLLABORATIVE LEARNING - PHYSICAL, TANGIBLE LEARNING - PLAYFUL EXPERIMENTS - PLAYFUL LEARNING - PLAYFUL USER INTERFACES - CONSTRUCTIVIST LEARNING

PROBLEM

We have been using our modern technologies every day for many reasons but when we consider young generations, it is hard to say that the modern interactivity of them is helpful for their physical and mental progress. Apparently, we need to define our current interactivity with these everyday devices by considering youngs and offer new collaborative, social and informative opportunities for their development.

How we take advantage of the technology for the education today?

Media Exposure on young people and it’s detrimental effects:

Cheap entertainment

Replacement of the hands-on activities

Passive consumption rather than involvement

Isolation from the outside

Misinformation

CONCEPTUAL QUESTIONS

How might we create playful learning experiences for collaborative, social learning in public spaces (schools, classes) by taking advantage of the technology?

How might we empower learners through playful experiences and together?

How might we use collaborative learning as a bridging the technology gap between children and families?

PROJECT GOALS

Give learners agency and autonomy in the experience, supporting them in taking an active role, rather than passive stance in their own learning

Support learners creative contribution to the learning material with the other learners

Support learners to use their bodies or the objects around them to contribute to the playfulness.

Engage learners with digital technologies and experiences that support “21st-century skills”

Let learners discover through play.

Support learners’ natural curiosity and social nature to play and learning

RESEARCH

Learning Styles

There are 7 primary learning styles:

Visual (Spatial) – These individuals learn best through pictures, images, and spatial understanding

Aural (Auditory) – These individuals learn best through sound and music

Verbal (Linguistic) – These individuals learn best through words, verbal and/or written

Physical (Kinesthetic) – These individuals learn best through experience and rely on the sense of touch

Logical (Mathematical) – These individuals learn best through logic and reasoning

Social (Interpersonal) – These individuals learn best through group interaction

Solitary (Intrapersonal) – These individuals learn best through self-study

Constructionist Learning Lab

● Learning by doing

● Technology as building material

● Hard fun

● Learning to learn

● Taking time - the proper time for the job

● You can’t get it right without get it wrong

● Do unto ourselves what we do unto our students

● Digital world

Social Learning Theory

Traditional theories of learning generally depict behaviour as the product of

directly experienced response consequences.

In actually, virtually all learning phenomena resulting from direct

experiences can occur on a vicarious basis through observation of other

people’s behaviour and it’s consequences for them.

In the social learning system new patterns of behaviour can be acquired

through direct experience or by observing the others.

PRECEDENTS

TSUMIKI is an interactive toy that combines building blocks with the digital world. The blocks' layout and positions are read by a sensor, and an image is projection-mapped onto the blocks to match how they are stacked. Depending upon how they are stacked, you can make animals appear or play number games.

RETAIL DESIGN EXPO DALZIEL&POW STANDS 1

Dalziel&Pow’s interactive wall is an interactive exhibition stand to demonstrate how appropriate technology can be used to engage and create shareable content. Our screen-printed illustrations sprung to life when touched, uncovering a host of playful animations, data-feeds, and sounds.

RETAIL DESIGN EXPO DALZIEL&POW STANDS 2

Dalziel&Pow’s interactive stand concept centres on interfaces activated and controlled by the audience. Visitors are invited to place a selection of everyday objects – a box, a plant, a smartphone, a coffee pot – from a shelf onto any of four sensors to trigger bespoke film content and accompanying soundtrack layers.

LOCAL PROJECTS BIO DESIGN STUDIO

At the Creature Creation Station, visitors create and test new virtual lifeforms by snapping flexible building blocks that represent DNA segments. Sensors in each block sync with the kiosks below to provide real-time feedback on the genetic design in progress. Without explicitly knowing it, visitors are creating a series of biological commands, with the DNA models serving as a proxy for genes.

PROTOTYPE

AUDIENCE AND CONTEXT OF USE

Who is it for?

For the primary school children and their families.

Where and when do they experience it?

At school or at the museum. I am planning to test it in public environment.

What are they doing while experiencing it?

They interact with the content and each other while learning by experiencing the content.

What the audience take away as the result of this experience?

Help people acquire new knowledge by linking it to the existing one and lived experiences.

They reach the conventional information in a more engaged way by using diverse senses and analytically understand their point of views among diverse perspectives.

POTENTIAL TECHNOLOGIES

Part1

https://github.com/mehtapaydin/OF_Week7/tree/master/Assignment_Week7_1

Part 2

Project Requirements

Sound: I am using 4 different sounds depending on the tribal rituals. Each sound has matched with diverse masks. When the user changes her facial gesture she also switches between sounds with the masks at the same time.

Movement & Easing: I am using Box2d vectors with gravity and controlling their movements with the facial expression. In the ideal scenario, my plan was to show diverse data oriented differentiations about each ritual on each side of the screen. So far I can control the movement and gravity but I can’t use data. I also used FBO to create a different effect but then I didn’t want to use it but it is still available in code. 

Drawing (images/shapes): I am using Masks as png images and OfBox2d Circles and Rectangles to be able to control with data.

Alpha Channels: I am using ofEnableAlphaBlending(); to create the same color both on the box2d objects and on Mask at the same time.

Fonts & Writing (a title screen at least): There is a title and 4 different ritual categories ( War, celebration, death, daily life), have been written on different sides of the screen to give the user more feedback about what are these rituals. And I am using two different typefaces:    

myFont1.load("FuturaPTMedium.otf", 20);    myFont2.load("BebasNeueBold.otf", 20);

An ofxAddon: I am using ofxFaceTracker to capture facial expressions, ofxCv and ofxOpenCv to detect head movements and keep the image on face accordingly. I am using ofxBox2d as data-oriented objects with ofxJSON. But unfortunately, I can not combine ofxBox2d objects with data.

A Web API: I was planning to use a museum API and my plan was to make a meaningful connection with box2d objects but I couldn’t make it since I can not control Box2d objects in the way I want.

A presentation: You can see the Final Presentation

A Blog post: http://mhtpydn.tumblr.com/post/158444651780/of-assignment-midterm-native-americans-tribal

Initial Idea and Project Goal

The tribal cultural tradition about body and face painting has always been very intriguing to me. At the beginning of the project, my idea was to create a mixed reality experience to give life this very old tradition today. After I started the project I had the chance to visit Brooklyn Museum and, I came across with brilliant Native American Masks at the exhibition names as “Life, Death, and Transformation in the Americas” and decided to create and installation about them. 

The main goal of the project is to create an immersive experience by using sound and personal gestures to navigate between 4 different masks including Celebration, War, Death and Daily life ceremonies. While doing this, I wanted the user to be able to get some feedback about some data belong to this terms but so far I couldn’t implement the data.

Inspirational Presentation

Native American Masks

Native American masks were an important part of the Native American culture. They were used during ceremonies, given as gifts, and each one was hand-crafted. They have a special charm that comes from the significance behind the art; the significance of the Native American way of life.

Native American masks were usually created in the likeness of an animal. These animals represented certain Native American ideas such as spirits, status, and characteristics. In addition, some Native Americans believed that each clan was descended from a different animal. This meant that some Native American masks related to the clan the Indians were from.

I chose four of them and categorized them according to the traditions such as death, war, celebration and daily life. Since it was difficult to find exact the same masks in a good jpeg quality I choose them based on my assumption and technical quality of the images. Then, I created the png files.

1- Native American Masks - Haida, Pacific North West

2- Native American Bird Mask, Tlingit people, Alaska

3- Bella Coola Komokwa Mask Kwaguilth, Native American

4-Mayan Mask

Precedents

Android Experiments - JONAS JONGEJAN & STEWART SMITH

Link

Golan Levin, “Messa di Voce”

Messa di Voce augments the speech, shouts, and songs produced by two virtuoso vocalists with real-time interactive visualizations.

A P P A R E L – Clothing in the age of data accumulation and machine learning

Link

Technical Research / Challenges

In order to implement my idea, I made a lot of research about using some of the Openframeworks Addons. OfxOpenCv, OfxCv, OfxFaceTracker and ofxBox2d were my focus points. Kyle McDonald’s libraries and especially the example projects available inside the repositories were my main resources to understand exactly how can I use and what is the limits of these add-ons. Also, Golan Levin’s assignments for Carnegie Mellon 

The main challenge for me to be able to run these examples on my computer. Because many of them were old and I needed to do some unique configurations to use them and using them took my whole 1 day. The second thins was to use face tracker add-on in a certain way. And lastly I tried to implement an API to use with box2d but I couldn't achieve it. 

Prototyping Process

Step1 

I tried to understand how can I use face tracker and simple gestures tracker. 

Step 2

I looked the example to use ofxCv and ofxOpenCv to be able to keep object and move them with the face. 

Step 3

I tried to add box2d and use them in accordance with the face expressions. 

Demo

Finally, I am using 4 face expressions resembling fear (War), smile(Celebration), close eyes (Death) and neutral(Daily Rituals). For each emotion and/or traditions there are four masks switching from one to another and there are 4 different ambient music which makes this experience more authentic.On the screen, there are box2d objects move from one edge to another to show realted data but I don’t have data yet. 

Github

Final Presentation

Further Improvements

I would like to implement a data and improve some of the problems related with the sound switch. Also, rather than using static masks I would like to manipulate mask based on talk and the other more advanced gestures. 

Resources

https://www.androidexperiments.com/experiment/giant-emoji

http://www.creativeapplications.net/openframeworks/a-p-p-a-r-e-l-clothing-in-the-age-of-data-accumulation-and-machine-learning/

https://www.brooklynmuseum.org/exhibitions/life_death_transformation

http://www.indians.org/articles/native-american-masks.html

https://anthropologylover.wordpress.com/2013/02/09/native-american-face-paint-customs-colors-designs/

Technical Resources:

https://www.freesound.org

https://github.com/kylemcdonald/ofxFaceTracker

https://github.com/kylemcdonald/ofxCv

https://github.com/vanderlin/ofxBox2d

https://github.com/kylemcdonald/ofxFaceTracker

Initial Concept

How might we create playful experiments by taking advantage of emerging technologies to teaching children basic tasks about real life within the future context?

How can we define playfulness in terms of daily experiments for the children?

How can we teach real life daily experiences (communication, friendship, physical realities) to children by using playfulness?

How might we create playful interfaces to convert digital dependence of today’s children into a benefit for them?

At the beginning of the project my I asked myself these questions in order to understand what is the next generation’s digital reality and how it will transform our everyday interactions;

What will be the interfaces of the next generation products and services?

In the future, how will be our relationship with our surroundings? 

How might we create new interactions with the objects/surfaces/spaces by merging our digital understanding today with the physical (real) world? 

How might we use dynamic smarter interfaces that can learn and adapt to diverse situations and us? 

Context

Learning

Children

Playful experiments

Playful learning

Playful User Interfaces

Digital learning

Self Learning

Playful learning

Initial Project Concept Schematics / Scenario

Audience/ Context of Us

Children and families those who use digital technologies are my audience. I want to make digital technologies to be used consciously. I also want to create technologies not to be a substitute for real experiences, instead, my goal is to create new experiences by taking advantage of digital concept to make the real life experiences more conscious and improved by using the technology. 

Precedents

Doodle Tunes

Andreas Refsgaard and Gene Kogan

Doodle Tunes lets you turn doodles (drawings) of musical instruments into actual music. A camera looks at your drawing, detects instruments that you have drawn, and begins playing electronic music with those instruments.

Sharing Faces – Seeing yourself reflected in the image of others

Kyle McDonald

This project was important to create new form of communication and visual empathy. 

Link1 

Lego Calendar by Vitamins Design syncs with Google Calendar

Vitamins Design Studio - London

A wall-sized calendar made entirely of Lego bricks that syncs with Google automatically via smartphone screenshots.

“It was quite an iterative process, trying to find the right balance between physical and digital,” says Adrian Westaway, the studio’s director of technology. “On one side you needed something big and clear that humans would like to use and look at–and on the other you needed something that was computer-readable and ordered.”

It offers a tactile way to keep track of the months ahead--and it syncs with Google Calendar via smartphone snapshots. 

Link1 

AR Music Kit

Yuri Suzuki

With the help of a few pieces of paper, AR Music Kit turns your device into a DIY musical instrument. When a code is presented to the camera, it plays back a sound. This provides a creative platform to create music in an unexpected and unusual way.

Link

Tag Me - An easy-to-use toolkit for turning the personal environment into an extended communication interface

MIT Media Lab

TagMe is an end-user toolkit for easy creation of responsive objects and environments. It consists of a wearable device that recognizes the object or surface the user is touching. The user can make everyday objects come to life through the use of RFID tag stickers, which are read by a RFID bracelet whenever the user touches the object.

Link 

Research Outline

Learning

Learning, critical thinking, interaction with the information.

Diverse learning methodologies.

The primitive experiences of playful learning.

Technology’s impact on information and learning.

Emerging ways of media usage by new generations.

Pros of digital interactions for learning / Real examples

Cons of digital interactions for learning / Real examples

Interviews, audience expectations.

Play / Playful Interactions

The future of interfaces and interactions.

The future of school, home, everyday interactions and the role of technology

The evolution of playful user interfaces

The future of user interfaces within the play context.

Technologies

Computer vision vs human vision, how can we interpret from automation of everyday tasks. 

Design Questions

What makes the object functional?

How important the emotional relationship of the people with the objects?

What makes these emotions possible?

Do the objects that we keep have some meaning,  a memory or our memories make them special?

Memories

How can we make these memories inherent in the object to show these emotions to each other?

Do the objects that we keep have some meaning,  a memory or our memories make them special?

Concept

Our idea is based on the memories that stay on the objects in both visible and invisible ways and we wanted to make them meaningful for the other people that come across with the same object by taking advantage of the digital techniques. We have chosen the book as the object to narrow down the experience into a single object and try to understand the signs of differentiating the meaning of differentiating experiences for each other. 

We are creating an interactive physical and digital experience with pages that contain different textures and objects. As the user flips through the pages of the book, he/she will come across all the traces of people who have used the book.

We want this book to be a representation of a book from a library so that it is reminiscent of understanding the memories and emotional connections others experienced while reading the book.

Precedents and Inspirations

Our first precedent was the palimpsests that investigate different layers of old books. The same experience can be seen in paintings as well. It was inspirational to see the signs of every stage or interactions with the artwork over the same paper. We started to think about the same idea for the everyday objects. 

The Old Guitarist (1903), Pablo Picasso

Codex Guelferbytanus, (VI Century)

Pentimento (2016), Claire Jeanine Satin

Another interesting project was about Fiona O’Leary’s experimental project about the usage of VR for everyday objects. She uses AR as a new interface for 3D objects and creates a kind of immersive new layer over the objects. In this example, it can be seen how she uses AR to show how much battery available for diverse content on a hard drive. It is quite interesting the way she leverage the physical experience in digital level. 

AR Harddrive (2016), Fiona O’Leary

Specifically after focusing on the book as an object, we found more precedents about the use of AR on book context.  

Augmented Book Prototype (2016), Paul Ferragut

Technology that we use

After all these research we decided to use Augmented Reality technology to show invisible memory layers on the book. 

The Project Narrative

According to our story, when people come across with a highlighted text or notes on the book, all these memories will be visible for the current reader withing the form of text or images. 

The second narrative was to show the content in a more interactive way by using video in AR. 

Protoype 1

Our first prototype was about creating text and image based memory interfaces over objects. It was our first sketch about VR before dicing into how to implement the technology. 

Prototype 2

Our second prototype is a book and we wanted to create different textures and tangible details to use together with AR screen. 

Prototype 3

Lastly, we implemented simple experiences in AR. We used Vuforia with Unity 3D to create surfaces that can be seen by computer within the unity. Then, we used some abailable 3D assets to use on this surface. Our idea was to create diverse interactions and memory pieces in different pages but for some technical problems, we were only able to implement limited experiences on the same texture. 

AR Test 1

AR Test2

Future Plans

We would like to try more AR features in Unity. It was the first time we used for this project, we learned a lot and we would like to implement more features for the next steps.

My Inspirations slides (Pecha Kucha) for the midterm assignment

https://docs.google.com/presentation/d/1g8H2YAW6fyxluS2wZzS73mSDz4BBiWRYzA2KlEETwAM/edit?usp=sharing

The Concept

The inspiration comes from mixing real-world images with the illustrations like it was in tribes. It is an old tradition to paint the body to express feelings or defending yourself from any kind of disaster coming from outside. After I research, I noticed that in the digital world there are many different examples showing us that it might be used to exaggerate the perception of a movement or an emotion on the canvas. So my plan is to create an audiovisual data collage showing human data relationships by taking advantage of simple shapes movements and human body or face and simple shapes. 

Perspective 1

Manipulating data with a tangible, physical sensors.

Perspective 2

Creating a game based interaction

Perspective 3

Making a projection mapping on human face or body.

Flocking Assignment Github Code