Helena's Professional Studies Blog

Collaborators:

Sanita Lustika provided help with creating the model of the ship in Maya. I would say she received 5/5 for her contribution and knowledge of Maya.

In return, I provided her with copywriting services, since her project required a lot of text. I made sure that all the text fitted with the tone and point, was easily understandable, and was grammatically correct.

Code:

This is the bullet creation code which finally utilizes all the audio data.

IEnumerator SyncToAudio(List<float>[] pitchData,List<float>[] timbreData, List<double> startData,List<float> duraData,double time){         yield return pitchData;         yield return time;

//waits for those to be received         audio.PlayScheduled(time+3);

//some leeway before playing, in case of lag. time represents when the audio is ready to play                           for (int k = 1; k < startData.Count; k++){             float[] noise=Array.FindAll(pitchData[k].ToArray(), (float x) => x>0.6);

//some filtering of data             int nois=noise.Length;                         while (AudioSettings.dspTime < time+0.7+startData[k]) yield return 0; // wait till the desired dsptime             for(int i =0; i< pitchData[k].Count; i++){                 if(pitchData[k][i]>0.99){

After the conclusion of the project, I think that I have successfully achieved the project goal of creating a music-centred shmup game. I took the initial concept and determined the crucial elements of it, planned out how I would achieve those features, and created a game which included all those essential features. However, I also selected some features which would be desirable for my game, but not essential. In terms of those features, I did not successfully incorporate them into the planning at the initial stages; they were left to the end of the plan, and I did not end up completing those features. So, although I was able to correctly prioritize the tasks I needed to complete, I did not effectively manage my development time to include features that, though they were not crucial, were still important. I also did not pay enough attention to the graphical side of the game, either during planning or during development. In addition, I did not properly manage finding contributors for those tasks such as graphics which I had problems with on my own; only the spaceship model was created by a contributor. In a way, the lack of contributors was not accidental; particularly in terms of coding, I felt like I wanted to understand the whole game creation process, and if I had someone else help with code that would not happen. However, choosing to share aspects of the creation process with others should have been part of the process itself, and I did not emphasize that enough.

User testing for this project consisted of having participants play a version of the game, recording the session using the Silverback software. Participants were asked to complete tasks in the game, including a full playthrough of a short level. Some also uploaded music to the game, though not their own music, but tracks already on the system. All participants provided useful feedback about their experience. Some of the feedback was taken into consideration when working on the next playable iteration of the game. One of the comments on earlier versions of the game was that the user found the large amount of bullets overwhelming, and they were unable to complete the goal of shooting the notes in time with the music, which gives a higher score. Therefore, when working on the next iteration of the game, what Patterns in Game Design refers to as “illusionary rewards” were added – a line representing the threshold a “note” passes when it occurs in the song, and a visible effect on the note to show when shooting it would increase the player's score multiplier. The colour of the bullet's explosions also changes when it is shot “in time”. With these additions, players were more easily able to select targets, and most could explain the purpose of the “threshold” line without being told.

At the beginning of the project's development, one of the initial tasks once the concept had been finalized was to decide on what features were “essential” and what features were “desired”. Once these lists had been made, it was possible to create a plan for the project, where the essential features were ordered and prioritized. It was apparent that certain features could only be completed after others were finished, so the plan was designed with these stages built-in. Once they were achieved, the next stage would immediately be moved on to – the task would only be completed as much as it needed to move on to the next stage, and then it would be focused on later and polished.

The first stage once concept research had been completed was to create an initial “proof-of-concept” style prototype, which would interpret the data of a single track. This did not include code enabling analysis of external music tracks.

Stage two was refining that prototype, by developing the music interpretation and basic gameplay. Once both were at a satisfactory level, stage three was the development of the capacity to retrieve and analyse a track selected by the user, and then apply the gameplay and interpretation developed in stage two. At this stage, the graphics not connected to the music data would begin to be developed.

After the initial research and planning, I had arrived at the concept of a music visualization game in the rhythm and shmup genres. I then progressed on to thinking about how to achieve my goal.

Since the concept was centred around music, I focused on that. The main feature was to make the user's music part of gameplay. Breaking this down, the game would need to first analyse the selected music track to get usable data, and then interpret this data as gameplay, and as other aspects of the game, such as the visuals; in other words, “What data can be gathered?” and “How can I use the data?”

The “analysis” and “interpretation” could therefore be seen as the two halves of the project, roughly corresponding to what happens “behind the scenes”, and to what the user actually plays and sees. Due to limitations of the Unity engine, tracks would be analysed via the external service “Echonest”. However, the results returned would have to be analysed still further, both during the development process and when the game is played. Lots of data is returned from the Echonest analysis, and while most of it is fairly well-documented, some aspects (such as the Timbre) are obscure and difficult to understand. After initial analysis, the next step is to create a basic interpretation of the data. The first interpretation was simply firing bullets when a musical “event”, such as a note being played, was detected. The bullet would be fired at a certain height based on the pitch of the note. However, once the data was interpreted in this way, I found that many pitches were detected during each event, resulting in a “wall” of bullets. After finding this, I went back to the analysis and examined ways of filtering the data to respond to only the strongest tones of the song. This required further analysis of the data, and when it was complete, the interpretation stage could be returned to again.

An important step forward in development was when I moved on from using a simple test track, and began to select more music tracks with many differences. The test track was fairly quiet, with a slow pace, so faster and more “aggressive” tracks were selected. This development was also a result of working on the code to send tracks for analysis, which originally had many problems; the test track was exported into Unity, and the analysis link was hard-coded in. After the system to select other songs was set up, the “interpretation” side also moved forward.

The development methodology for this project meant that there was usually a large technical challenge for each iteration. The main issues were mostly due to the technical limitations of the Unity platform. One of the crucial features of the project was the ability to use the user's music to create gameplay. This requires the use of Unity's "streaming audio" feature to retrieve the user's selected track. It also requires the music to be analysed for data. However, Unity did not seem to have the ability to analyse the data of streamed audio. To get around this problem, I decided to use an external service to analyse the audio, "Echonest" (Echonest 2014), which also provided other detailed data which Unity did not, such as Timbre. Though this solved the immediate problem, it required dealing with external data, and it meant the game would need an internet connection to run. As makeuseof (and many other similar articles) states: “Many new mobile games require an Internet connection to play... It’s unfortunate because there are times when it’s just not an option” (Patkar 2014). Since one of my “desired” features was a mobile version, and even a desktop application can be played on a laptop in similar situations, I felt this was an important consideration.

No other essential feature required an internet connection, so I decided to keep its use to a minimum. Instead of connecting to Echonest to retrieve the analysis each time the level was played, I implemented a method to save the data to the computer after the first analysis, requiring a connection only when analysing. If no analysing is done, the game can be played in areas without internet connection. Using the example in the makeuseof article, a user could analyse several tracks before an aeroplane flight, and then not need internet access on the plane.

Other problems I came across were ones tangential to the main goals of analysing and interpreting, such as interpreting the JSON data of the analysis, and implementing a visual interface to allow the user to select files. These are both complicated problems which I did not want to focus on; therefore, I used code that people had provided for those purposes, LitJSON (2013) and ImprovedFileBrowser (Brauer 2012). Especially when dealing with the file system of the user's computer, making sure errors do not occur is crucial, and these few external scripts were well constructed and already had any errors ironed out. This enabled me to focus on the gameplay and working with the audio data.

One technical issue where the solution was not implemented was with the streaming audio feature in Unity. Due to licensing issues, Unity only allows streaming MP3s, the most common audio file type, on mobile platforms, which I was not developing for (Unity 2013). I had investigated ways of using MP3s in the project, but since audio files in the ogg format could be streamed, I focused development on using the data from tracks I could already stream, and considered implementing MP3 support as a “desired” feature instead of an “essential” one. If this project had been a commercial one, I would also have been required to pay a licence for the use of MP3s.

Another crucial technical factor was makingsure the timing of the events was always consistent with the audio. In the initial prototype, the time in samples that the track had been playing for was used to time directly based on the audio track, since simply counting in frames or seconds could be imprecise, and any slowdown in the game could lead to gameplay events being out of sync. This method worked well; the times given for audio events in the analysis would be translated into samples, and when the current sample matched, the code attached to it would fire.

However, this initial prototype used audio that had already been imported in to Unity. When the project progressed to using audio selected from the file system and streamed in to the game, I found that Unity could not access the samples of the audio, so they could not be used for timing.

After searching, I found a method which someone had used to synchronize events to the rhythm of audio, though it does not directly use the audio data to synchronize with (Floisand 2014). The “dspTime” variable worked perfectly when I adapted my code to use it. It also enabled me to get around the slight delay between the streamed audio being ready to play and actually playing, by adding a countdown.

Combating possible slowdown was an important part of the game design, not just in terms of code, but also visual design. The graphical style of my game is very simple; only one model was created for the game, and that model was also very simple. The only other objects are the bullets, which are simple spheres. None of the objects have complicated textures. In addition to having to load less assets, the graphical simplicity of the gameplay is due to the fact that many bullets will be thrown at the player during gameplay, and unlike traditional “bullet hell” shooters, the bullets will not be produced in strict patterns which make them easier to comprehend. Having the bullets be simple, with a dark background as a contrast to their bright, changeable colours means the bullets can be easily seen, and so should be less overwhelming (Shmuptheory 2010). In addition, the colour and length of the bullets is dependent on the audio data, and since that data can have a wide range of values, I decided that applying changes to a simpler object would work better than a complex one which may not fit well with certain variables. The display of the score and rhythm is also simple and out of the way of the action, so it isn't distracting.

Shmuptheory., 2010. The Anatomy of a Shmup. Available from: http://shmuptheory.blogspot.co.uk/2010/02/anatomy-of-shmup.html[Accessed 23 May 2014]

Unity, 2014., Audio Files. Available from: https://docs.unity3d.com/Documentation/Manual/AudioFiles.html[Accessed 23 May 2014]

LitJSON., 2013. LitJSON. Available from: http://lbv.github.io/litjson/[Accessed 23 May 2014]

Brauer, D., 2012. ImprovedFileBrowser. Available from: http://wiki.unity3d.com/index.php?title=ImprovedFileBrowser[Accessed 23 May 2014]

Echonest, 2014., Dev Center. Available from: http://developer.echonest.com/[Accessed 23 May 2014]

Floisand, C., 2014. Beat Synchronization in Unity. Available from: http://christianfloisand.wordpress.com/2014/01/23/beat-synchronization-in-unity/[Accessed 23 May 2014]

Music-based games are enjoyed by a wide audience. More "traditional" music games such as Guitar Hero have had widespread appeal, due to the popular songs included. Games such as Audiosurf expand the music library to any song the user chooses. This expands the potential audience to players with wide-ranging music tastes, and Audiosurf immediately became a very successful and popular game on the Steam platform. (Boyer 2008)

Score-based games, such as both rhythm games and shmups, are "score-based" games. Both rhythm games and shmups consist mainly of what Patterns in Game Design refers to broadly as “rhythm-based actions”: “Actions that require players to time their actions several times in a row.”

In rhythm games, these timing cues correspond to the music, and in shmups, they correspond with when the target is directly in front of the player. This type of gameplay in rhythm games often does not include severe penalties such as a game over, but the loss of score or combos. This opens the gameplay up to players of any level - anyone can get the satisfaction of completion, but the score goal still invites comparison, and gives players a measure of "game mastery" (Bjork and Holopainen 2005, p.357), or the distinction “between skilful and incompetent players when they are using all their skills and abilities in a game.”

Without a game-ending penalty for bad play, any player can enjoy the game, no matter their skill level. Shmups tend to include game-over conditions, and many, such as the Touhou Project series of bullet hell shooters, are extremely difficult, which can be off-putting for the more casual gamers, such as those who traditionally play rhythm games (Stewart 2011). Audiosurf does not end the level if a player plays badly, and in this case I think that model is a good one to mimic.

Looking at potential players of rhythm games using the “unified player personality styles” model (Stewart 2011), the type of gameplay I'm aiming for in this project would appeal to both the “achiever”, who would be motivated to play the track repeatedly for a high score, and the “socializer”, who would simply enjoy experiencing their music in a different way and discussing it with others. A large part of Audiosurf's popularity is likely due to the socializers, since players of Audiosurf are encouraged to post videos of their gameplay online, as well as “compete with others on the internet for the high score on your favorite songs.”(Audiosurf 2010)

Achievers can focus on goals such as high scores as well as goals such as completing a difficult level, and socializers may be more casual players who reject the barrier to entry provided by high difficulty. Therefore, I concluded that there should be no great obstacles to the player's survival, but that the gameplay should remain complex enough to accommodate “mastery”. This suggested the inclusion of a skill-based combo system which would provide high score rewards for more dedicated players.

Bjork, S. and Holopainen, J., 2005. Patterns in Game Design. Hingham, MA: Charles River Media, Inc.

Boyer, B., 2008. Audiosurf Tops February Steam Sales. Available from: http://www.gamasutra.com/php-bin/news_index.php?story=17735[Accessed 23 May 2014]

The concept of my project, "Synesthesia", is a music visualisation game in the gameplay genres of "rhythm" and "shmup". A rhythm game involves the gameplay responding to the music playing in the game. Examples include Guitar Hero, Audiosurf, Harmoknight and Theatrhythm Final Fantasy. A shmup, or 2D shooter, is a genre where the gameplay “takes place in an environment viewed from either a top-down or side-view perspective... In many of these games the player is under attack by overwhelming numbers of enemies and must shoot them as fast as possible”. (Adams 2010, p.393)

Examples of shmups are the Darius series, Space Invaders, R-Type, and the Touhou series.

“Synesthesia” uses a side-view perspective, mimicking the bars of musical notation.

The audio, visuals and gameplay are all directly connected to each other; with the audio forming the direct basis for the two other aspects.

The wider project goal was to manage the development process of a game, from the concept development to the realization of that concept. This includes the planning, research, time management and user testing of the project.

Since the audio visualisation angle suggests a focus on user-provided content - i.e. the user's own selection of music tracks - the technical and design aspects will centre around procedurally generated content, as opposed to pre-designed elements; for example, instead of creating specific levels, creating a method which can create many levels, given the user-provided content. This removes the need to design levels, a task not intended to be the focus of the project, whilst simultaneously providing an almost infinite amount of levels, which do not need background music to be created or licensed; the decline of Guitar Hero was partly to do with the costs of licensing popular and well known music (Wired 2011), costs which aren't needed when the user provides the music.

Since the project was centred around music, I first looked at the patterns used in rhythm games, such as Guitar Hero and Audiosurf. According to Patterns in Game Design (Bjork and Holopainen 2005, p.364), for “Rhythm-Based Actions” which correspond with “non-dangerous activities”, there is usually “feedback to how well players are executing the actions”, which is provided by indicators and rewards which do not “affect the actual gameplay”. It also describes a possible consequence of “failing to keep the rhythm” as ending “extended actions” such as combos. These patterns led me to decide on a score-based gameplay system for my game: the only goal would be achieving a good score, so all penalties would be tied to the score. With this in mind, I decided that the levels would not have a drastic failure consequence such as a game-over. A drastic consequence, when combined with levels generated from data with a wide range of possibilities, could lead to the player becoming frustrated by being unable to complete a level, since they are not pre-designed. When the goals become score-based, the difficulty of a track does not matter, since the measure of “game mastery” would only be through comparing the scores achieved on that single track.

Having decided on a score-based gameplay system, I wanted to expand the idea. I focused on the connection between audio and visuals, one of the key concepts. Comparing the visuals of many game genres, I found that the most similar to the patterns of a music visualization were a certain type of “shmup”, the “bullet hell” shooter. An article on Rhizome (Bailey 2013) describes the visuals of what they refer to using the japanese term “danmaku” (bullet curtain) as where “...orb- or arrow-shaped projectiles form the atomic units of pulsating, multi-hued latticeworks and arabesques. Sometimes these designs will manifest as screen-swallowing circles with equidistant radii or spokes...”.

The article suggests that these games are played as much for the “bewildering” aesthetics of the bullet patterns as for the challenging gameplay. Comparing images of music visualizations and bullet hell shooters shows similarities in their patterns, so the many brightly-coloured bullets of this type of game seemed a perfect fit for objects to be generated from music data.

The player goals of a shmup also commonly include score-based goals.

With this chosen, I began to integrate many of the patterns of a shmup into my concept. The Fundamentals of Game Design (Adams 2010, p. 393) states that “The action in 2D shooters takes place in an environment viewed from either a top-down or side-view perspective.”

Having looked at the top-down style, I researched games which used a side-on view. Once again, I compared them to music visualisations, and I also compared them to rhythm games. Here I also found similarities in layout. The music related visualisations/games are mimicking the layout of a musical score, with the notes displayed at different vertical heights depending on pitch. Although the side-scrolling shooter is not based on the layout of musical notation, the musical layout could easily be adapted into it.

Another phrase I noticed from the article about danmaku games (Bailey 2013) is that “Arcade gamers must learn very quickly... to view these tantalizing clusters of glowing globules or phosphorescent spear tips as the primary focus of their visual attention”.

In rhythm games, the primary focus of attention is the notes themselves – they are what is acted upon by the player. In bullet hell games, the player ends up focusing on the bullets due to their visuals. And since the concept involves using the notes of the track as bullets, I decided I should combine the ideas, and make the bullets themselves the target.

Looking at this idea after user testing, and after the completion of the game, I think it works well for slower, simpler tracks, which were what was used in user testing as well as development. Faster tracks with more notes may be more overwhelming, though there is not the frustration of a game-over.

Bjork, S. and Holopainen, J., 2005. Patterns in Game Design. Hingham, MA: Charles River Media, Inc.

Adams, E., 2010. Fundamentals of Game Design. Second Edition. Berkeley, CA: New Riders.

Bailey, T.B.W., 2013. The Danmaku Game as a New Optical Art, Part 1. Available from: http://rhizome.org/editorial/2013/feb/7/danmaku1/[Accessed 23 May 2014]

Wired, 2011. Activision Bails out of Guitar Hero, Cancels Games. Available from: http://www.wired.com/2011/02/guitar-hero-canceled/[Accessed 23 May 2014]

Ruckman, J., 2006. Whitney Music Box Variations. Available from: http://jruck.us/post/41109046184/whitney-music-box-variations[Accessed 23 May 2014]

Our fifth and final lecture was given by Richard Wallis, and was on the subject of the producer's relationship with the client.

One of the main points of the lecture was the idea of 'Convergence Culture', in particular the way it changes how the producer and client work together. As Henry Jenkins (2006 p. 5) states, “the emerging convergence paradigm assures that old and new media will interact in ever more complex ways”. What this means is that the previously defined roles involved in production are blurring; there is crossover between 'market areas'. And since much of this converged content is filtered through various interactive media, an interactive media practitioner may end up working with clients from many different fields. This particularly holds true for web developers, since almost every organisation or individual in every field will want some kind of web presence.

So when pitching for a client, there are many fields it will be important to have knowledge of – this actually mirrors something said in a production seminar this term. Thorough research must be done to ensure that both producer and client can understand each other correctly, especially if the pitch is successful. In this case, Richard advised that communication is essential. He placed emphasis in the idea of 'managing expectations', so the client will not be dissappointed.

If this approach is compared with Evan Grant's approach above, Evan's approach seems like it would be a better environment to work in. However, I recognise that the approach outlined by Richard is the more realistic one, and even in a situation with a pre-existing concept or prototype that can be demonstrated to work, a client-centred approach can also be needed.

The fourth lecture, presented by Evan Grant from seeper, was a little more abstract than the others; the focus was not on a specific topic within professional studies, but on a specific company's work. However, the lecture as included thoughts from Evan on being part of a research and development-oriented company, rather than simply a commercial one.

The projects presented here fit in very well wil our 'no user interface' brief in the production unit. They provide a great example of how even some of the more wild-sounding concepts in that area can actually be produced. To me, they also show how important it is to think creatively and produce something different from the usual screen-based interface. Evan suggested that by not filtering the experience through a screen, things become more intuitive, and the user is closer to the “essence of experience”. I think this “direct feedback and connecting between senses” chimes quite well with Andy Clark's (2003 p.28) ideas about “transparent technology”. In fact, he states that “Well-fitted transparent technologies have the potential to impact what we feel capable of doing, where we feel we are located, and what kinds of problems we find ourselves capable of solving.”. So creating intuitive technologies that become a 'part' of us can have a large effect.

Evan Grant also talked about what it is that makes intuitive experiences like the ones he creates so compelling. He explained that it is the 'magic' of an experience that draws people in – the sense of wonder that an experience inspires. Specific parts of something will get picked up on and remembered, and will become the 'hook' that not only attracts people, but also defines how they explain the experience to others.

Furthermore, he also explained how this can, in addition to attracting audiences, attract investors and work partners. He also said that this kind of collaboration, where the partner comes to you, can maintain the integrity of the project. By creating the project first, it is already defined and shown to work, and the ability of people to alter it is lessened.

However, for someone who can't afford to create something without the funding being there already, this approach would be very difficult. But Evan also mentioned that one place where you are more free to develop what you choose is in university.

So in conclusion, I should make the most of my time here and try and use my resources to the fullest.

The third subject specific lecture was on the topic of copyright, and was presented by Kris Erickson and Bartolomeo Meletti. Since copyright is a more 'objective' topic, I learned a lot of concrete information about how copyright and intellectual property work. For example, one of the major issues raised was the difference in copyright between countries. Since the internet and many other forms of interactive media are international, this often creates conflicts, where someone may inadvertently break the law despite being assured that work is out of copyright. And copyright for a single work can also often be split across the work's different aspects, such as soundtrack, recording, and video. This further complicates things, as it means that more information needs to be found to determine legality.

However, I also learned that there are many helpful resources that provide media that is free of copyright, and also that there are exceptions to copyright – the one applying to me being the 'research and study' exception, which will of course expire when my course finishes. It might be a good idea to make full use of this whilst I still can, but I think that it is still a good idea for me to check copyright, and also to try not to use copyrighted work, since it is a good habit to get into for when the exception no longer applies.

The second lecture was delivered by Jo Lush, and the topic was the creation of digital projects in an agency environment. There was a lot of focus on the 'roles' that members of an agency might have. Of the roles she defined, many have aspects that interest me. I actually found doing user testing interesting for my production work, and I also find considering how users might choose to interact with a product to be interesting, so usability specialist might be viable. I also find the more creative aspects, such as designing, to be quite fulfilling. Production also appeals to me, so that could be an option as well. Of course, I will have to specialise if I wanted to fit into one of the roles described. Fortunately, I think this course provides enough experience in all those areas that I could make an informed decision about specialising.

The presentation was also focused on the process of creating a product; specifically, the 'waterfall' method. This method seems to be quite rigid; all the stages have to be completed exactly in order. In addition, user testing is restricted to the conclusion of the project. My experience with user testing for my production portfolio has been that it is extremely useful to user test at early stages, since there is very likely to be a problem with how users interact with a product, and if key features have serious problems here, it could be crippling. It's for these reasons that some lectures in the Production unit viewed the waterfall approach as being inadequate. However, this lecture showed that there are upsides. The rigid structure works better in an agency with many people, since it means that each person should know exactly what they should be doing now, and the next stage they are working towards, so many people can all be on the same page in terms of production. The structure and timescale can also work to ensure that a project will be completed in reasonable time.

The first subject-specific professional studies lecture was given by Mike Hawkyard of the company 4T2. His lecture seemed to have a focus on the tactical choosing of projects, or in other words the importance of being able to say 'no' to projects. An idea that was returned to throughout the presentation was the idea of having a goal in mind for your career, and taking only jobs which will bring you closer to that goal. This was expanded upon when he advised to choose content with a specific next step – a project where you could say with confidence the type of job you will aim for afterwards. However, whilst I can see this being a good idea for an established company, or even an already-successful freelancer, I am not sure how well it would serve a company which has just started, or an inexperienced freelancer. If there is only a very few options available in terms of projects, it would be hard to refuse an opportunity.

However, that doesn't mean that no projects at all should be refused. Mike also brought up the idea of creating a checklist with scored criteria. A job can be examined under those criteria and given a score based on what criteria it fulfils. Different questions can have different scores, depending on their importance. I can see this as a useful tool over the course of a career – as you gain more experience and reputation, the score that must be fulfilled before a job is accepted can be increased, or perhaps the weighting of the criteria can be altered.