Just putting the finishing touches to my dissertation in preperation for a final revision.
over 35 pages
over 10,000 words
looking forward to a five minute break.
Monday, 18 May 2009
Tuesday, 31 March 2009
Reset
If you think you need to reset something, then it's highly likely that the library has a reset call in it.
DiracReset(true, m_pDiracSystem);
Doh!
DiracReset(true, m_pDiracSystem);
Doh!
Sunday, 29 March 2009
Cue The Music
I ran into a slight hiccup earlier in the week when I realised one of two possible problems; either my timing calculation algorithm produced unavoidable rounding errors or Reason 3 was not writing files out with exactly the right amount of PCM samples in them. This resulted in any track that was the full length of the sequence going out of time with the click track, which is not good.
No matter which of these reasons were causing this the solution seemed the same - Add some silent PCM samples at the end of the audio sample to make up the 0.01 percent (guesstimate) missing, or time stretch the sample by the small amount to account for the missing samples. The second solution was my first choice as the first could introduce some pops/clicks in the sound if the discrepancy was large, although if the discrepancy is too large then that will cause other issues of timing. As the DIRAC time stretching library had already been integrated it was just a case of finding the percent of missing samples and process the sound to the new length. This worked a charm and now my sequences can loop indefinitely.
With the timing and the layering systems all functioning correctly it was now time to move on to the audio cues. In general there are two different roles that a cue can perform: alter the playback of a given sequence; provide a transition between sequences. To help keep track of the various different cues I use these two basic roles as separators. A cue manager is used to create each instance of a cue so that a record of it can be stored, to allow access to the cue at a later date. Here is a diagram of how the cue creation process is structured.
In this structure the cue manager uses a template Factory Method to instantiate cues of varying types, ensuring that the data type at least inherits the AudioCue class by only returning pointers to objects of that data type. For recording purposes two STL maps are used to store pointers to objects that are of type "SequenceCue" or "TransitionCue". As stated before, these types define the two distinguishing roles of any cue.
The foundation of these cues is based on the virtual method "Perform". Every concrete class must override this method in order to allow the cue to manipulate a sequence or set of sequences in some way. Structure wise this design is similar to the Command pattern, in the way that the core of the design is based around an "execute" method. Now when calling a cue the application need not know what concrete type of cue it is calling, only that it is either a SequenceCue or a TransitionCue - the overridden "Perform" method handles all the intricate logic related to the cue.
There can be three methods of triggering a cue: on a beat; on a bar; instantly. On a beat or bar could signify a specific number in the sequence or just the next beat or bar that will be played. This is where the click track proves useful here, as the callback from it instantly (instantly when we call system::update ... those darn synchronous callbacks) tells us when we've hit a beat or bar, and from that we can start an activated cue.
The design structure of the cue system keeps it highly modular, allowing for easy addition of new cues that could incorporate features such as standard DSP effects or additional sounds to be layered in the sequence.
No matter which of these reasons were causing this the solution seemed the same - Add some silent PCM samples at the end of the audio sample to make up the 0.01 percent (guesstimate) missing, or time stretch the sample by the small amount to account for the missing samples. The second solution was my first choice as the first could introduce some pops/clicks in the sound if the discrepancy was large, although if the discrepancy is too large then that will cause other issues of timing. As the DIRAC time stretching library had already been integrated it was just a case of finding the percent of missing samples and process the sound to the new length. This worked a charm and now my sequences can loop indefinitely.
With the timing and the layering systems all functioning correctly it was now time to move on to the audio cues. In general there are two different roles that a cue can perform: alter the playback of a given sequence; provide a transition between sequences. To help keep track of the various different cues I use these two basic roles as separators. A cue manager is used to create each instance of a cue so that a record of it can be stored, to allow access to the cue at a later date. Here is a diagram of how the cue creation process is structured.
In this structure the cue manager uses a template Factory Method to instantiate cues of varying types, ensuring that the data type at least inherits the AudioCue class by only returning pointers to objects of that data type. For recording purposes two STL maps are used to store pointers to objects that are of type "SequenceCue" or "TransitionCue". As stated before, these types define the two distinguishing roles of any cue.
The foundation of these cues is based on the virtual method "Perform". Every concrete class must override this method in order to allow the cue to manipulate a sequence or set of sequences in some way. Structure wise this design is similar to the Command pattern, in the way that the core of the design is based around an "execute" method. Now when calling a cue the application need not know what concrete type of cue it is calling, only that it is either a SequenceCue or a TransitionCue - the overridden "Perform" method handles all the intricate logic related to the cue.
There can be three methods of triggering a cue: on a beat; on a bar; instantly. On a beat or bar could signify a specific number in the sequence or just the next beat or bar that will be played. This is where the click track proves useful here, as the callback from it instantly (instantly when we call system::update ... those darn synchronous callbacks) tells us when we've hit a beat or bar, and from that we can start an activated cue.
The design structure of the cue system keeps it highly modular, allowing for easy addition of new cues that could incorporate features such as standard DSP effects or additional sounds to be layered in the sequence.
Friday, 20 March 2009
Not Strictly Asynchronous
In order to keep a low overhead for using FMODex the developers consciously decided to not make it thread safe. While this is nice if you want to squeeze the most out of your audio code, the design of the API has left me with a slight issue relating to my click track implementation.
Callbacks are easy to set up in FMOD, but they do not work how you expect them to. In order to trigger a channel callback you must first call the FMOD system method "update". In fact, to update any of the 'asynchronous' features of FMOD you require to call system::update(), and this method must be called in the same thread that all other FMOD commands have been called in because of the non-thread-safe nature of the API.
For a simple while loop:
while (true)
{
FmodSystem::Update();
}
Everything runs fine, and to a certain extent (depending on how time sensitive you are) the method of using a click track would be enough to keep everything in time without the need for sample accurate sound stitching. However, Once an element of delay is introduced - Sleep(30) - timing discrepancies begin occurring with the triggering of samples. With a delay of Sleep(100) the timing is really poor; with a delay of Sleep(1000) it's a mess. This is fairly obvious when you think about it as the callbacks would only be triggered every 1000 milliseconds or so, and when a beat generally occurs approximately every 0.3 seconds in a 4/4, 120 bpm sequence, you can see where the delay is coming from.
I am going to try handling the system::update() in its own thread to see if this can keep the callback timings consistent, but I will need to be careful not to access sensitive FMOD methods from this new thread.
Callbacks are easy to set up in FMOD, but they do not work how you expect them to. In order to trigger a channel callback you must first call the FMOD system method "update". In fact, to update any of the 'asynchronous' features of FMOD you require to call system::update(), and this method must be called in the same thread that all other FMOD commands have been called in because of the non-thread-safe nature of the API.
For a simple while loop:
while (true)
{
FmodSystem::Update();
}
Everything runs fine, and to a certain extent (depending on how time sensitive you are) the method of using a click track would be enough to keep everything in time without the need for sample accurate sound stitching. However, Once an element of delay is introduced - Sleep(30) - timing discrepancies begin occurring with the triggering of samples. With a delay of Sleep(100) the timing is really poor; with a delay of Sleep(1000) it's a mess. This is fairly obvious when you think about it as the callbacks would only be triggered every 1000 milliseconds or so, and when a beat generally occurs approximately every 0.3 seconds in a 4/4, 120 bpm sequence, you can see where the delay is coming from.
I am going to try handling the system::update() in its own thread to see if this can keep the callback timings consistent, but I will need to be careful not to access sensitive FMOD methods from this new thread.
Monday, 16 March 2009
Old Ideas Return
When breaking up a piece of music into various layers that play independently a major concern is how to keep these layers in time with each other. Wwise achieves this by making sure that each layer is of the same sample length. This is also how I got round the issue during Dare To Be Digital. The whole point of this project, however, is to allow for layers to be of varying lengths. So how can the synchronisation of these layers be achieved?
The first method I looked into simply involved counting the beats. If you have a sequence that you know the tempo and time signature you can use that to work out how long, in milliseconds, a single beat should be. A timer can then be used to count the milliseconds between beats.
While in theory this is a "sound" idea (sorry, but I had to get that pun in eventually) issues can occur with synchronising the start of the timer with the start of the sequence. A separate thread created by the FMOD api handles playback of the sounds meaning that the timer is being calculated outwith the sound playback. With the time-critical nature of audio, and particularly music, this method is inevitably too unpredictable to confidently use, as the timer will always be separate from the source.
After realising the pitfalls of using a timer I then returned to my original idea of using an audio sample as a "click track". The click track was based on a sample containing a single bar of audio that had markers placed on the beats. During playback of this sample when the playhead passed one of the markers a callback would be triggered indicating that we've hit a beat. With this method the timing is kept in the same thread as the playback of the audio, so timing discrepancies are reduced.
This method was used in the Dare To Be Digital game, Origamee, to synchronise the audio cues with the beats of the music. However, it didn't work properly as it suffered from the delay of looping static sounds, so the cues would go out of time. Luckily as the tempo for the game music was fast it was hard to spot the timing issues with the cues, and because the music layers were the same length the music itself wouldn't go out of time. Now that I can implement sample accurate stitching this can be used to resurrect the click track, as it overcomes the issue with the loop delay.
The first method I looked into simply involved counting the beats. If you have a sequence that you know the tempo and time signature you can use that to work out how long, in milliseconds, a single beat should be. A timer can then be used to count the milliseconds between beats.
While in theory this is a "sound" idea (sorry, but I had to get that pun in eventually) issues can occur with synchronising the start of the timer with the start of the sequence. A separate thread created by the FMOD api handles playback of the sounds meaning that the timer is being calculated outwith the sound playback. With the time-critical nature of audio, and particularly music, this method is inevitably too unpredictable to confidently use, as the timer will always be separate from the source.
After realising the pitfalls of using a timer I then returned to my original idea of using an audio sample as a "click track". The click track was based on a sample containing a single bar of audio that had markers placed on the beats. During playback of this sample when the playhead passed one of the markers a callback would be triggered indicating that we've hit a beat. With this method the timing is kept in the same thread as the playback of the audio, so timing discrepancies are reduced.
This method was used in the Dare To Be Digital game, Origamee, to synchronise the audio cues with the beats of the music. However, it didn't work properly as it suffered from the delay of looping static sounds, so the cues would go out of time. Luckily as the tempo for the game music was fast it was hard to spot the timing issues with the cues, and because the music layers were the same length the music itself wouldn't go out of time. Now that I can implement sample accurate stitching this can be used to resurrect the click track, as it overcomes the issue with the loop delay.
Friday, 30 January 2009
DIRAC Time Stretching & Pitch Shifting
Whilst the majority of the recent work has involved a lot of planing and structuring of the tool application and its code base, I have also spent some time investigating the time stretching technique I intend to implement. Over the past few days it has become clear that I am unlikely to succeed in producing a time stretching algorithm capable of the tasks I would deligate to it. There are so many existing algorithms out there, each fairly complicated, that produce results of varying quality depending on the audio signal fed into them. For example the Rabiner and Schafer method does not work so well for polyphonic sounds but the phase vocoder method can introduce an audible after-effect known as "phase smearing".
However, I have come across a C/C++ library specially dedicated to pitch shifting and time stretching known as DIRAC. It comes in various flavours but their completly free DIRAC LE version interested me the most. This still provides the quality that you get with the studio and pro levels but puts some limits on things like supported sample rates and number of channels which don't really affect me too much.
DIRAC was developed by Stephan Bernsee, the founder of Prosoniq, and it can be seen in many professional level products such as Steinberg's Wavelab and Nuendo. It is an incredibly small and simple library containing a total of 7 functions and 3 enumerations, however the quality is astounding for it being a free product ( let's pretend we didn't see the 9870 EURO price tag for the full version ).
After a couple of days of tinkering I've managed to take a sound that's been loaded into FMOD and stretch/squash it using DIRAC. My test program is still a bit flaky though as I'm still not 100% sure how the sound buffers work. Doesn't help when FMOD works in bytes and DIRAC works in samples, but I'll get there.
However, I have come across a C/C++ library specially dedicated to pitch shifting and time stretching known as DIRAC. It comes in various flavours but their completly free DIRAC LE version interested me the most. This still provides the quality that you get with the studio and pro levels but puts some limits on things like supported sample rates and number of channels which don't really affect me too much.
DIRAC was developed by Stephan Bernsee, the founder of Prosoniq, and it can be seen in many professional level products such as Steinberg's Wavelab and Nuendo. It is an incredibly small and simple library containing a total of 7 functions and 3 enumerations, however the quality is astounding for it being a free product ( let's pretend we didn't see the 9870 EURO price tag for the full version ).
After a couple of days of tinkering I've managed to take a sound that's been loaded into FMOD and stretch/squash it using DIRAC. My test program is still a bit flaky though as I'm still not 100% sure how the sound buffers work. Doesn't help when FMOD works in bytes and DIRAC works in samples, but I'll get there.
Friday, 23 January 2009
Updating ...
It has been a while since the last post, partly due to needing a little time to relax but primarily because of the updates needed to be made to my team's Dare To Be Digital 2008 entry - a winning entry if you can believe that. Extra work was needed to be done to it in preparation for the BAFTAs in March, however, now that is out the way I can get down to business again with this project.
With the final submission of my project proposal I narrowed down two main features to focus on that I felt were most neglected in current game audio tools. These features are time stretching and sample accurate looping.
Time Stretching
This is the method of lengthening or shortening an audio sample while preserving its pitch. During my research I have not found a single game specific application that supports this feature meaning that if any samples were to be layered together as part of one track they would have to be edited in another package first before they could be used together if their tempo's differed. In a review of the tool Wwise, Michael Henein stated that he would like to see time stretching implemented in these kind of tools to allow for more automated syncing with musical cues. I agree with his statement as tools are supposed to aid the developer - it's no use if you have one tool but are quite restricted in how you use it unless you have access to others.
In this project the time stretching will be implemented in a similar fashion to how Sony Acid has used it. With Acid, when you load in a sample you can make it fit with the rest of the project your working on by making its tempo match the projects tempo by either stretching or shortening it. This feature in a game specific tool would allow the audio designer/composer to just dump any suitable samples into their application without having to worry about their pacing as the tool will be able to make them all fit together.
Time stretching can have some adverse affects on certain types and lengths of samples such as the creation of artifacts ( clicks and pops ) and just making it sound unplesant. This is where the judgment of the user will be required to assess if more drastic measures are needed such as recording a new sample or changing the layout of the track.
Sample Accurate Looping
Unfortunatly with the audio APIs I have looked into I discovered ( through experience ) that when you set a sound to loop there is a slight delay from when it ends to when it loops round to the start. While this delay is inaudable it does pose an issue where multi-layered, different length sounds are concerned. The reasoning behind this was explained this previous post.
Why not just use samples that are the same length when layering them I hear you ask? Well, if you have ever taken a look at a complicated musical composition in any sequencer based program ( Cubase, Logic, Reason, Acid etc.) you'll notice that little variations, short phrases and contained loops can be found all over the place, seperated by blank spaces. If you were to create a single loop with samples of the same length then each sample would have to be as long as the longest sample. Not only would this take up huge amounts of unnecessary memory/CPU overhead in playing silent passages it also restricts how dynamic the composition can be. Individual phrases or loops could never be randomly played at any time unless an extra layer was added just for those bits - a layer that would again have to be the same length as the longest sample.
With using sample accurate looping, sections that have a lot of silence would simply not be played, reducing memory footprint as well as CPU overhead. Compositions could also be made more dynamic by allowing the composer to define sections that can have variations played in them, of which a certain variation could be picked in real time depending on the game state.
With the final submission of my project proposal I narrowed down two main features to focus on that I felt were most neglected in current game audio tools. These features are time stretching and sample accurate looping.
Time Stretching
This is the method of lengthening or shortening an audio sample while preserving its pitch. During my research I have not found a single game specific application that supports this feature meaning that if any samples were to be layered together as part of one track they would have to be edited in another package first before they could be used together if their tempo's differed. In a review of the tool Wwise, Michael Henein stated that he would like to see time stretching implemented in these kind of tools to allow for more automated syncing with musical cues. I agree with his statement as tools are supposed to aid the developer - it's no use if you have one tool but are quite restricted in how you use it unless you have access to others.
In this project the time stretching will be implemented in a similar fashion to how Sony Acid has used it. With Acid, when you load in a sample you can make it fit with the rest of the project your working on by making its tempo match the projects tempo by either stretching or shortening it. This feature in a game specific tool would allow the audio designer/composer to just dump any suitable samples into their application without having to worry about their pacing as the tool will be able to make them all fit together.
Time stretching can have some adverse affects on certain types and lengths of samples such as the creation of artifacts ( clicks and pops ) and just making it sound unplesant. This is where the judgment of the user will be required to assess if more drastic measures are needed such as recording a new sample or changing the layout of the track.
Sample Accurate Looping
Unfortunatly with the audio APIs I have looked into I discovered ( through experience ) that when you set a sound to loop there is a slight delay from when it ends to when it loops round to the start. While this delay is inaudable it does pose an issue where multi-layered, different length sounds are concerned. The reasoning behind this was explained this previous post.
Why not just use samples that are the same length when layering them I hear you ask? Well, if you have ever taken a look at a complicated musical composition in any sequencer based program ( Cubase, Logic, Reason, Acid etc.) you'll notice that little variations, short phrases and contained loops can be found all over the place, seperated by blank spaces. If you were to create a single loop with samples of the same length then each sample would have to be as long as the longest sample. Not only would this take up huge amounts of unnecessary memory/CPU overhead in playing silent passages it also restricts how dynamic the composition can be. Individual phrases or loops could never be randomly played at any time unless an extra layer was added just for those bits - a layer that would again have to be the same length as the longest sample.
With using sample accurate looping, sections that have a lot of silence would simply not be played, reducing memory footprint as well as CPU overhead. Compositions could also be made more dynamic by allowing the composer to define sections that can have variations played in them, of which a certain variation could be picked in real time depending on the game state.
Wednesday, 19 November 2008
Proposal Presenation
This week saw the entire year presenting their individual topics to a panel of lecturers in order to get their projects approved. Out of the few presentations I witnessed there were certainly some interesting, well thought out topics. This post is to sum up my proposal presentation in order to clarify my project's area of focus, as it has changed subtly throughout this blog so far.
Here is an e-rendering of my presentation based on my notes written before hand. It went pretty much as it is written:
" Hi, I'm Jonathon and my project is on developing adaptive music for video games "
"Why developing adaptive music? Well, I have experience in both the technical and artistic elements involved in developing audio for computer games. More notably, I have successfully implemented a basic adaptive music system for the recent Dare To Be Digital competition. This was based on mixing different layered tracks of a song in real-time depending on the main characters status."
"The issues surrounding this topic stem from a constant, growing demand placed on sound designers. This growing demand requires excellent, well established tools to give the content creator more freedom and control. Thus increasing their productivity."
"The aim of the project is to determine where current audio middle-ware has not advanced far enough to meet the demands of the content creator and, if possible, provide new alternatives and improvements. Audio is a cast discipline that contains many elements out with the scope of this project so the focus is purely on the music."
*recite research question*
"To address this question I intend to: Analyse existing audio tools, including applications not specific to video game audio. Determine what the sound designer/composer needs in such tools.
Develop an underlying framework that can support these requirements. Produce a suitable user interface to allow non-programmers to manipulate the framework effectively."
Any Questions? (there was another slide but didn't feel the need to upload the image)
Here is an e-rendering of my presentation based on my notes written before hand. It went pretty much as it is written:
" Hi, I'm Jonathon and my project is on developing adaptive music for video games "
"Why developing adaptive music? Well, I have experience in both the technical and artistic elements involved in developing audio for computer games. More notably, I have successfully implemented a basic adaptive music system for the recent Dare To Be Digital competition. This was based on mixing different layered tracks of a song in real-time depending on the main characters status."
"The issues surrounding this topic stem from a constant, growing demand placed on sound designers. This growing demand requires excellent, well established tools to give the content creator more freedom and control. Thus increasing their productivity."
"The aim of the project is to determine where current audio middle-ware has not advanced far enough to meet the demands of the content creator and, if possible, provide new alternatives and improvements. Audio is a cast discipline that contains many elements out with the scope of this project so the focus is purely on the music."
*recite research question*
"To address this question I intend to: Analyse existing audio tools, including applications not specific to video game audio. Determine what the sound designer/composer needs in such tools.
Develop an underlying framework that can support these requirements. Produce a suitable user interface to allow non-programmers to manipulate the framework effectively."
Any Questions? (there was another slide but didn't feel the need to upload the image)
...
Luckily the only major question asked was from my technical supervisor. Unfortunately I did not answer it as elegantly as I had hoped. The question was along the lines of "what do you mean by data-driven?". I cannot begin to replicate the ramblings of my original answer but for those who are interested here is a better thought out response.
"What do I mean by data-driven?"
"In essence the term data-driven is referring to how the framework is to be used. Once integrated into the game engine the framework will no longer require the use of a programmer to define functionality. All the rules and data will be gathered together by the sound designer/composer in the tool which will output information to control the framework. In other words, the "data" gathered in the tool is used to "drive" the framework."
So, to summarise, my project is about finding out as much as I can regarding current audio middle-ware that has some bearing on developing adaptive music. Once enough information has been gathered then it should be easy to ascertain where the game specific tools lack functionality found in the more mature audio applications. I will attempt to implement a few of these aspects in a simple tool of my own to show how they can make developing adaptive music easier.
"What do I mean by data-driven?"
"In essence the term data-driven is referring to how the framework is to be used. Once integrated into the game engine the framework will no longer require the use of a programmer to define functionality. All the rules and data will be gathered together by the sound designer/composer in the tool which will output information to control the framework. In other words, the "data" gathered in the tool is used to "drive" the framework."
So, to summarise, my project is about finding out as much as I can regarding current audio middle-ware that has some bearing on developing adaptive music. Once enough information has been gathered then it should be easy to ascertain where the game specific tools lack functionality found in the more mature audio applications. I will attempt to implement a few of these aspects in a simple tool of my own to show how they can make developing adaptive music easier.
Saturday, 15 November 2008
Coffee?
From my understanding of music composer setups, a Mac DAW ( Digital Audio Workstation ) is as common as a Windows based one. This suggests that the end user targeted by this project could be on either platform and developing for one will completely cut out the other. In an area of development where helping the user is the primary focus, isolating one group because of platform choice might not be in the best interests of the application. Perhaps C# is not the ideal language to develop this software with as using windows forms will restrict users to a windows environment.
A different, more cross platform approach would be to use the JVM environment. Java has a very powerful GUI development API called Swing, rivaling Microsoft's Windows Forms and after following some of this tutorial on using the JNI ( Java Native Interface ) I have learned that it is fairly simple to call C++ code in Java similar to the way I can in C#. So far Java would act in a similar fashion to C# in this project while hopefully providing a platform independent solution. The only conflicts that would arise regarding platform dependence would come from the unmanaged C++ code. These would be possible to resolve as the audio API of choice, FMOD, is cross platform. From what I can tell from the Firelight website is that they're are different libraries depending on run time environment so this may be an issue if the underlying C++ structure will need to be tailored for the specific platform.
While Java does not have any support for pointer arithmetic ( a large advantage of C# ) the JNI does allow java objects to be manipulated in unmanaged code. This does pose as a huge advantage over C# because even though you can marshall C++ data types to C# data types you cannot do it the other way round. Obviously without a complete design layout of the application I cannot determine which techniques are more appropriate to this situation. However, at face value Java does seem to be very inviting.
Some tests and tutorials will need to be completed before a final decision can be made. I do have experience with C# but having never touched Java before it would be foolish to jump straight in and depend on it.
A different, more cross platform approach would be to use the JVM environment. Java has a very powerful GUI development API called Swing, rivaling Microsoft's Windows Forms and after following some of this tutorial on using the JNI ( Java Native Interface ) I have learned that it is fairly simple to call C++ code in Java similar to the way I can in C#. So far Java would act in a similar fashion to C# in this project while hopefully providing a platform independent solution. The only conflicts that would arise regarding platform dependence would come from the unmanaged C++ code. These would be possible to resolve as the audio API of choice, FMOD, is cross platform. From what I can tell from the Firelight website is that they're are different libraries depending on run time environment so this may be an issue if the underlying C++ structure will need to be tailored for the specific platform.
While Java does not have any support for pointer arithmetic ( a large advantage of C# ) the JNI does allow java objects to be manipulated in unmanaged code. This does pose as a huge advantage over C# because even though you can marshall C++ data types to C# data types you cannot do it the other way round. Obviously without a complete design layout of the application I cannot determine which techniques are more appropriate to this situation. However, at face value Java does seem to be very inviting.
Some tests and tutorials will need to be completed before a final decision can be made. I do have experience with C# but having never touched Java before it would be foolish to jump straight in and depend on it.
Thursday, 13 November 2008
References and Bibliography
This post is going to act as a place to store all references to any reading material etc. with regard to the project. It shall be expanded upon when new material is consulted.
Material
Halliwell, L. 2008. Building World-Class Art Tools. [Online] Available at:
http://lukehalliwell.wordpress.com/category/game-development/tools/ [ Accessed 16th November 2008 ]
Material
Anderton , C., [no date]. So What Does "Sample Accurate" Mean, Anyway? [Online]. Available at: http://www.harmony-central.com/articles/tips/sample_accuracy/ [accessed 17th October 2008]
Audiokinetic. 2007. Wwise [software] version 2008.3. Available from http://www.audiokinetic.com/4105/try-wwise-now.asp [ Accessed 19th October ] Montreal, Quebec, Canada : Audiokinetic inc.
Barrier, M. Interview with Carl Stalling [ online ] Available from: http://www.michaelbarrier.com/Funnyworld/Stalling/Stalling.htm [ Accessed 27th April 2009 ]
Bernsee, S. Time Stretching And Pitch Shifting of Audio Signals – An Overview. [ online ] Available from: http://www.dspdimension.com/admin/time-pitch-overview/ [ Accessed 20th October 2008 ]
Bersnstein, D., 1997. Creating an Interactive Audio Environment. [Online]. Available at: http://www.gamasutra.com/view/feature/3238/creating_an_interactive_audio_.php [accessed 16th April 2009]
Brandon, A. 2004. Audio for Games: Planning, Process and Production. New Riders.
Brandon, A., 2008. Next-Gen Audio Square-Off: PlayStation 3 vs. Xbox 360. [Online]. Available at:
http://www.gamasutra.com/view/feature/3814/nextgen_audio_squareoff_.php [accessed 1st October 2008]
Chang, K. S., Kim, G. B., Kim, T. Y. 2007. Video Game Console Audio: Evolution and Future Trends. Washington: IEEE Computer Society. Available from : http://doi.ieeecomputersociety.org/10.1109/CGIV.2007.87[ Accessed 3rd November 2008 ]
Chuck D, 2003. Composing, Producing and Implementing an Interactive Music Soundtrack for a Video Game, lecture at the Game Developers Conference, San Jose, 2003.
Chuck, W., Dawson, B. 2006. Coding for Multiple Cores, lecture at the Game Developers Conference, San Jose, 2006.
Clark, A., 2007. Defining Adaptive Music. [Online]. Available at: http://www.gamasutra.com/view/feature/1567/defining_adaptive_music.php [accessed 1st October 2008]
Creative Labs. 2008. OpenAL [online]. Available from: http://connect.creativelabs.com/openal/default.aspx [Accessed 2nd December 2008]
Firelight Technologies Pty. Ltd.. 2008. FMOD [online]. Available from: http://www.fmod.org/
[Accessed 2nd December 2008]
Halliwell, L. 2008. Building World-Class Art Tools. [Online] Available at:
http://lukehalliwell.wordpress.com/category/game-development/tools/ [ Accessed 16th November 2008 ]
Henein, M. 2007. Sound Integration. Mix, Professional Audio and Music Production [online]. Available from: http://mixonline.com/recording/applications/audio_next_revolution_played/ [Accessed 11 November 2008]
Henein, M. 2007. Field Test: Audiokinetic Wwise 2007.1 Videogame Software. Mix, Professional Audio and Music Production. [online]. Available from: http://mixonline.com/gear/reviews/audio_audiokinetic_wwise_videogame/ [ Accessed 19th October 2008]
Hoffman, K. 2006. Microsoft Visual C# 2005 Unleashed. SAMS publishing.
Khawaja, N., 2008. Making Your Game Tools Fast And Efficient . [Online]. Available at: http://www.gamasutra.com/view/feature/3798/making_your_game_tools_fast_and_.php [accessed 16th October 2008]
Kirmse, A., 2004. Game Programming Gems 4 . Charles River Media.
LucasArts. 1991. Monkey Island 2: LeChuck's Revenge [Disk]. Amiga, MS DOS, Macintosh, FM Towns. LucasArts.
Novell Group. 2008. Mono [software] Version 2.0. Available from : http://www.go-mono.com/mono-downloads/download.html [accessed 16th November 2008]
O'Donnell, M., 2002. Producing Audio For Halo. In: Game Developer Conference, San Jose, 24 March, 2002 [Online]. Available at: http://halo.bungie.org/misc/gdc.2002.music/ [accessed 1st October 2008]
Parker, J. R., Heerema, J. 2007. Musical Interaction in Computer Games. New York: AMC. Available from http://portal.acm.org/citation.cfm?id=1328243 [Accessed 3rd November 2008 ]
Patin, F. Beat Detection Algorithms. [online]. Available at: http://www.gamedev.net/reference/articles/article1952.asp [accessed 1st september]
Robinson, C., et al. 2007. The Twelfth Annual Interactive Music Conference Project BAR-B-Q 2007. Group Report: Overcoming Roadblocks in the Quest for Interactive Audio. Texas, San Antorio. Available at http://www.projectbarbq.com/bbq07/bbq07r6.htm [ Accessed 1st October ]
Schildt, H. 2006. C++ The Complete Reference. 4th edition. California: McGraw-Hill/Osbourne .
Shah, S. Playstation 3 Next-Gen Audio: Metal Gear Solid 4. [Online]. Available from http://uk.gamespot.com/features/6147812/index.html [Accessed 16th April 2009 ]
Sony Creative Software Inc.. 2008. ACID Pro [software] Version 7.0. [ online ] Available From : http://www.sonycreativesoftware.com/download/trials/acidpro [ Accessed 1st December 2008 ]
Sterns, B. Trail: Java Native Interface. [ online ] Available From : http://java.sun.com/docs/books/tutorialNB/download/tut-native1dot1.zip [Accessed 15th November 2008 ]
Whitmore, G., 2003. Design With Music In Mind: A Guide to Adaptive Audio for Game Designers . [Online]. Available at: http://www.gamasutra.com/view/feature/2852/design_with_music_in_mind_a_guide_.php [accessed 1st October 2008]
WikiRecording. Digital Recording. [online]. Available from: http://www.wikirecording.org/DAW [Accessed 2nd December 2008]
Tuesday, 4 November 2008
Proposal Draft
Been a couple of weeks since I updated this but i was interupted by the London Games Festival. Needless to say Gears of War 2 is superb.
Anyway.
Honours project proposal draft - it is away and awaiting peer review. Included in the proposal is a brand new research question based upon something put together in a meeting with my technical supervisor. Between us, Henry and I managed to produce this :
"What methods could be used to design an audio framework which can be used to structure/organise audio samples to facilitate interactive manipulation within a game envionment?"
While encorporating all the desired elements this question is still not very elegant. Through further filtration the question has become this : (in the proposal)
"What methods could be used to design a data driven audio framework for the purpose of structuring and organising audio samples in order to create flexible video games scores?"
Now we're getting there. The idea of being data driven ties into the tool aspect while defining the game scores as flexible explains how the result is adaptive.
I found a perfect book for this topic on Safari. Audio For Games: Planning, Process and Production by Alexander Brandon. However Safari only allowed Abertay users to view a preview of this book. The university library does have two hard copies but they are both out on loan at the moment. Requested one for recall but I fear that I might not get it till it's too late. Certainly won't recieve a copy in time for writing my final proposal.
In closing I must report that this blog will soon not be officially used by the university to check up on progress. We are getting new "lecturer friendly" spaces to post our findings. However I will not abandon this blog as I have become quite attached to it! So anything put up on the new one will find its way on to here in some form or another.
Anyway.
Honours project proposal draft - it is away and awaiting peer review. Included in the proposal is a brand new research question based upon something put together in a meeting with my technical supervisor. Between us, Henry and I managed to produce this :
"What methods could be used to design an audio framework which can be used to structure/organise audio samples to facilitate interactive manipulation within a game envionment?"
While encorporating all the desired elements this question is still not very elegant. Through further filtration the question has become this : (in the proposal)
"What methods could be used to design a data driven audio framework for the purpose of structuring and organising audio samples in order to create flexible video games scores?"
Now we're getting there. The idea of being data driven ties into the tool aspect while defining the game scores as flexible explains how the result is adaptive.
I found a perfect book for this topic on Safari. Audio For Games: Planning, Process and Production by Alexander Brandon. However Safari only allowed Abertay users to view a preview of this book. The university library does have two hard copies but they are both out on loan at the moment. Requested one for recall but I fear that I might not get it till it's too late. Certainly won't recieve a copy in time for writing my final proposal.
In closing I must report that this blog will soon not be officially used by the university to check up on progress. We are getting new "lecturer friendly" spaces to post our findings. However I will not abandon this blog as I have become quite attached to it! So anything put up on the new one will find its way on to here in some form or another.
Friday, 17 October 2008
Sample to Sample Accuracy
In audio terms a sample refers to a single element of an audio signal. It contains a value and a duration to represent that signal at that given moment. This Harmony Central article explains the need for sample to sample accuracy in great detail.
To summarise it quickly I have put together a simple example.
The image above shows two audio signals, one on top in green and one on the bottom in yellow. Lets say these signals play together as a section of a song, the top being the drums and the bottom being the bass, continuously looping. It is obvious from the image that the bottom signal is shorter than the top but if you look at the timeline it is actually only shorter by five hundredths of a second - inaudible!
While not noticeable on its own, after these samples have looped a hundred times that discrepancy (shown by the red ) will become one millisecond. Slowly the more each signal loops they will grow further and further out of time. If the signals had the exact same amount of samples then they would NEVER go out of time. This is why sample to sample accuracy is required when dealing with looping audio.
To summarise it quickly I have put together a simple example.
The image above shows two audio signals, one on top in green and one on the bottom in yellow. Lets say these signals play together as a section of a song, the top being the drums and the bottom being the bass, continuously looping. It is obvious from the image that the bottom signal is shorter than the top but if you look at the timeline it is actually only shorter by five hundredths of a second - inaudible!
While not noticeable on its own, after these samples have looped a hundred times that discrepancy (shown by the red ) will become one millisecond. Slowly the more each signal loops they will grow further and further out of time. If the signals had the exact same amount of samples then they would NEVER go out of time. This is why sample to sample accuracy is required when dealing with looping audio.
Wednesday, 15 October 2008
Programming Progress
Over the past couple of days I've been tinkering with some programming, trying to get to grips with certain processes. Since I decided to look into tool development for this project I turned to C# and windows forms. I had a bash with them and it was so easy to knock some stuff together - the new guide system that helps you position your elements on the form is fantastic!
I know some stuff about C# but to develop any sort of audio application I really want to use FMOD as I'm so comfortable with it. There is an FMOD wrapper for C# but my familiarity with it lies in C++. After having a glance over FMOD# and its appalling lack of documentation I started to go off the idea of using C# at all until I read about using unmanaged code in it (Visual C# Unleashed 2005). This book explained how to load an unmanaged DLL into C# which would allow me to write my audio engine completely in C++, abstracting all but the necessary data which could be accessed through the exported DLL functions.
Sounded too good to be true so I began to do some simple tests. While I learned the basics quickly these simple tests didn't really give much to work with so I decided to take my growing networking module code and adapt it round a DLL wrapper. This was perfect. The code was complicated enough to give a good result (multiple threads, dynamic memory allocation etc.) and didn't need any extra work so I could get straight into C#.
So I defined my DLL export functions, loaded it into C# and Bobs your uncle, we have C++ being called from C#. I even managed to figure out how to debug in mixed mode so I could see what the values were in the C# and C++ code in the same project. All good things must come to an end though as I soon discovered a threading problem. Windows forms are very picky about threads. From what I've discovered, any thread you want to run in a windows form that requires access to UI elements needs to be created by the thread that made them ( see here ). This is also known as a single threaded apartment environment.
I deduced that this must be related to the thread I was running in my DLL to check for received packets over the network. I was wrong obviously as this thread doesn't need access to the forms ui. It turned out to be an issue with using BeginInvoke instead of just Invoke in the C# check-for-new-packets thread. I don't quite understand yet why BeginInvoke causes a deadlock until a new packet is received as all the research I've done on the two seems to indicate that it shouldn't. Hopefully I will find the answer someday.
I know some stuff about C# but to develop any sort of audio application I really want to use FMOD as I'm so comfortable with it. There is an FMOD wrapper for C# but my familiarity with it lies in C++. After having a glance over FMOD# and its appalling lack of documentation I started to go off the idea of using C# at all until I read about using unmanaged code in it (Visual C# Unleashed 2005). This book explained how to load an unmanaged DLL into C# which would allow me to write my audio engine completely in C++, abstracting all but the necessary data which could be accessed through the exported DLL functions.
Sounded too good to be true so I began to do some simple tests. While I learned the basics quickly these simple tests didn't really give much to work with so I decided to take my growing networking module code and adapt it round a DLL wrapper. This was perfect. The code was complicated enough to give a good result (multiple threads, dynamic memory allocation etc.) and didn't need any extra work so I could get straight into C#.
So I defined my DLL export functions, loaded it into C# and Bobs your uncle, we have C++ being called from C#. I even managed to figure out how to debug in mixed mode so I could see what the values were in the C# and C++ code in the same project. All good things must come to an end though as I soon discovered a threading problem. Windows forms are very picky about threads. From what I've discovered, any thread you want to run in a windows form that requires access to UI elements needs to be created by the thread that made them ( see here ). This is also known as a single threaded apartment environment.
I deduced that this must be related to the thread I was running in my DLL to check for received packets over the network. I was wrong obviously as this thread doesn't need access to the forms ui. It turned out to be an issue with using BeginInvoke instead of just Invoke in the C# check-for-new-packets thread. I don't quite understand yet why BeginInvoke causes a deadlock until a new packet is received as all the research I've done on the two seems to indicate that it shouldn't. Hopefully I will find the answer someday.
Worksheet 2 Explained
I like the way this year is set up in regards to peer review. It really helps you to refine an idea when you get so much feedback. On the other hand, giving your own feedback to others certainly opens the mind letting you think "outside the box".
So what was worksheet 2 about? In worksheet 2 I had to formulate a research question for my project and detail how I would go about answering it. Now, surprisingly (or perhaps unsurprisingly, depending on your age and outlook) adaptive music in computer games has definitely been around a long time. Looking back to what people have produced not just in the industry but also in previous years of CGT I have decided it would be wise to approach this subject indirectly. Still keeping the theme of adaptive music but not having that as the only focus. Without further delay, here is my second initial question ( first was rubbish :P )
So what was worksheet 2 about? In worksheet 2 I had to formulate a research question for my project and detail how I would go about answering it. Now, surprisingly (or perhaps unsurprisingly, depending on your age and outlook) adaptive music in computer games has definitely been around a long time. Looking back to what people have produced not just in the industry but also in previous years of CGT I have decided it would be wise to approach this subject indirectly. Still keeping the theme of adaptive music but not having that as the only focus. Without further delay, here is my second initial question ( first was rubbish :P )
"What aspects present in traditional audio engineering applications can be tailored for use in tools specifically designed for developing adaptive music in computer games?"
... say what?
Let's run through it.
"What aspects present in traditional audio engineering applications ... ". Skipping the aspects portion for a minute let us look at the traditional audio engineering applications bit. What is a "traditional" audio engineering application? Back in the day, audio engineers would use giant mixing desks, complicated systems of wiring and controllers etc. but with modern processing power the average joe can simulate this with a digital audio workstation. These workstations would run all the hardware related tasks of audio engineering but in software. According to Wikipedia this concept was made popular in 1987 with a package created by Digidesign that later went on to become the industry standard, Pro Tools.
What aspects of applications such as Pro Tools, Sound Forge, ACID pro etc. am I talking about? There are lots of features in these programs that make them successful (too many to look into, list or even think about), so which ones would benefit the development of adaptive music? It would be silly to try and recreate something like Logic Audio, considering that it has been around for years, gone through many iterations and is incredibly stable. These applications are generally used for music generation and because of their long running development are very effective at this. Leading me to believe that computer game development tools tend not to be a replacement for these programs but more of an extension, trying to make the content to engine pipeline a lot smoother. People still use Max and Maya, even though they are not solely geared towards games. So to ascertain what aspects of audio engineering applications would benefit the development of adaptive music a look into current game audio tools is needed.
There are a fair few tools out there for game audio design, although from what I've seen very few actually "create" the music from scratch (extension of existing applications, remember). One that I hope to single out for my project is a tool mentioned in the previous post, Wwise. Wwise is a fantastic design tool that once the programmer has integrated the framework into the engine the audio content creator has full control over every part of the audio in game. Wwise is a complete audio design tool but the majority of it will be disregarded as I am only interested in the adaptive music section. Here you can load in wavs, structure them, assign their playback to game triggers, control their volume etc. all really cool stuff. Aside from assigning playback to game triggers this functionality is very familiar to programs such as ACID pro, Pro Tools, Sound Forge. The crunch here is what can tools like Wwise learn from these well developed applications to make producing adaptive music more accessible/efficient?
I cannot say exactly at this moment in time what would or wouldn't benefit tools like Wwise but from personal experiences here is what I feel is a drawback. Currently, any sample brought into Wwise must be of the same length as any other sample that it is to loop with to ensure sample to sample accuracy. Reason3 certainly does not have this problem. As long as the samples are in the same time signature, bbm and their length is in multiples of a bar length then you're good to go. Surely this should be able to work in Wwise. ACID pro bypasses the bbm constraint by dynamically time stretching the sample to fit and Pro Tools can even bypass the time signature by dynamically finding the beats and moving them! Surely these aspects could be put to good use in adaptive audio development as they would remove a fair amount of constraints on the composer. These are just personal examples that I feel are important but obviously more research will need to be completed before a more educated example can be given. There might be a very good reason for the sample to sample accuracy :D
Let's run through it.
"What aspects present in traditional audio engineering applications ... ". Skipping the aspects portion for a minute let us look at the traditional audio engineering applications bit. What is a "traditional" audio engineering application? Back in the day, audio engineers would use giant mixing desks, complicated systems of wiring and controllers etc. but with modern processing power the average joe can simulate this with a digital audio workstation. These workstations would run all the hardware related tasks of audio engineering but in software. According to Wikipedia this concept was made popular in 1987 with a package created by Digidesign that later went on to become the industry standard, Pro Tools.
What aspects of applications such as Pro Tools, Sound Forge, ACID pro etc. am I talking about? There are lots of features in these programs that make them successful (too many to look into, list or even think about), so which ones would benefit the development of adaptive music? It would be silly to try and recreate something like Logic Audio, considering that it has been around for years, gone through many iterations and is incredibly stable. These applications are generally used for music generation and because of their long running development are very effective at this. Leading me to believe that computer game development tools tend not to be a replacement for these programs but more of an extension, trying to make the content to engine pipeline a lot smoother. People still use Max and Maya, even though they are not solely geared towards games. So to ascertain what aspects of audio engineering applications would benefit the development of adaptive music a look into current game audio tools is needed.
There are a fair few tools out there for game audio design, although from what I've seen very few actually "create" the music from scratch (extension of existing applications, remember). One that I hope to single out for my project is a tool mentioned in the previous post, Wwise. Wwise is a fantastic design tool that once the programmer has integrated the framework into the engine the audio content creator has full control over every part of the audio in game. Wwise is a complete audio design tool but the majority of it will be disregarded as I am only interested in the adaptive music section. Here you can load in wavs, structure them, assign their playback to game triggers, control their volume etc. all really cool stuff. Aside from assigning playback to game triggers this functionality is very familiar to programs such as ACID pro, Pro Tools, Sound Forge. The crunch here is what can tools like Wwise learn from these well developed applications to make producing adaptive music more accessible/efficient?
I cannot say exactly at this moment in time what would or wouldn't benefit tools like Wwise but from personal experiences here is what I feel is a drawback. Currently, any sample brought into Wwise must be of the same length as any other sample that it is to loop with to ensure sample to sample accuracy. Reason3 certainly does not have this problem. As long as the samples are in the same time signature, bbm and their length is in multiples of a bar length then you're good to go. Surely this should be able to work in Wwise. ACID pro bypasses the bbm constraint by dynamically time stretching the sample to fit and Pro Tools can even bypass the time signature by dynamically finding the beats and moving them! Surely these aspects could be put to good use in adaptive audio development as they would remove a fair amount of constraints on the composer. These are just personal examples that I feel are important but obviously more research will need to be completed before a more educated example can be given. There might be a very good reason for the sample to sample accuracy :D
Wednesday, 8 October 2008
Time to Wwise up
So I guess the old saying is true - there definitely is NOT anything new under the sun.
Over the past week I have discovered that there are already many middle-ware programs that aid in creating adaptive music for the sole use in computer games. Surprisingly this can be traced back as far as the early 90's with LucasArts' in house engine iMuse. Although strictly Midi based in its infancy, iMuse enabled the audio designers at LucasArts to create incredibly complicated music structures with intricate transitions linking them. See here for interviews and details on iMuse.
Returning to the here and now, there is a new audio programming API offering a similar service as FMOD that was released in 2006. This API, known as Wwise, has one of the most in depth middle ware tools for video game music creation I have seen so far. With enough time and resources you could certainly create music that is dynamic enough to conform to what was described in the previous post. However, is the amount of time required to do this too much? Is this why games still have those cringworthy audio disasters? Not enough time?
Perhaps these tools still have a little way to go before developers will be able to get the most out of them in a constrained time frame.
Over the past week I have discovered that there are already many middle-ware programs that aid in creating adaptive music for the sole use in computer games. Surprisingly this can be traced back as far as the early 90's with LucasArts' in house engine iMuse. Although strictly Midi based in its infancy, iMuse enabled the audio designers at LucasArts to create incredibly complicated music structures with intricate transitions linking them. See here for interviews and details on iMuse.
Returning to the here and now, there is a new audio programming API offering a similar service as FMOD that was released in 2006. This API, known as Wwise, has one of the most in depth middle ware tools for video game music creation I have seen so far. With enough time and resources you could certainly create music that is dynamic enough to conform to what was described in the previous post. However, is the amount of time required to do this too much? Is this why games still have those cringworthy audio disasters? Not enough time?
Perhaps these tools still have a little way to go before developers will be able to get the most out of them in a constrained time frame.
Wednesday, 1 October 2008
Just back up! Back up! ... mind your business that's all ...
Received my first piece of peer feedback today. It gave the impression that I may have not explained clearly what this project is to entail ... so I begin this post with a slight step back. And a question:
"What is adaptive music?"
This article by Andrew Clark goes into some detail about how to define adaptive music. However, to effectively grasp this I am going to take a stab at how I perceive the concept.
The interest in computer game audio for me began with one of those early classics (early for me anyway :p) Final Fantasy 7. I loved that game. From the story, to the graphics, to the music, the whole experience drew me in to the make believe world.
When I first bought the game I had been spending time at my grans for a holiday. Ten years old and completely oblivious to pretty much everything. So I get the game, get into the living room where my Playstation was set up and start playing. Needless to say I played that game non-stop throughout that entire holiday and loved every minute of it. My parents however were not as enthusiastic. The main reason - "that awfully repetitive music".
Being only ten I thought it was just "the oldies" showing their age. How could the music in this awesome game be repetitive? I didn't notice anything, it was great! I did however go back and have a play of FF7 a year or 2 ago and damn... that's some repetitive music. Now that my ear has had another ten years to develop I begin to notice what my parents were on about. Sure the music changed when you had a battle, went to a different town or one of the major characters dies ( everyone must have played it by now ) but the music didn't just change, it was more a case of one track ending and another track beginning. This exact situation can be replicated quite easily by loading up your favourite mp3 player and occasionally skipping tracks. Certainly not adaptive music.
So far I took a dig at a game which is over ten years old - ten has seemed to crop up quite a bit in this post - and admittedly games that take the same approach are few and far between when "next-gen" is considered. That's not to say there are games out there that don't do it. Titles like Tony Hawks, Pure and Madden are exempt from this discussion as their music is generally comprised of a playlist containing popular, chart type singles. The games I'm targeting are the types of game that when you play it you're drawn in by either story or a specific game mechanic. Examples of these games are Gears of War, Dawn of War and God of War ( is it just me, or do gamers like war? ).
So, what is adaptive music? What I believe adaptive music to be is a form of playing music in a game that is as dynamic as say, an AI controlled bot in Counterstrike... or the procedural animation used in The Force Unleashed. The music is so unpredictable that even the composer is not too sure what the music will do next, but you can be sure that whatever it does will make sense and not be jarring to fellow audiophiles like myself.
The only effective example of this must be taken not from games, but from film. A similar media but definitely much more predictable and easily demonstrated as far as score is concerned. For this we turn to our pal Indy. Notice the subtle changes in the music to reflect the on screen action. Excellent examples of this are when he takes his gun out to find that it's empty, when "dad" gets the idea to chase the birds and when the focus rests on Indy at the end of the clip the main theme is effortlessly slipped in. This is the kind of subtlety that is missing in modern day computer games as far as in game music is concerned.
That's not to say that games are not trying to be more like films in terms of score, and call me stubborn but I don't think they are doing the beautiful music composed for them justice. I'll quickly draw upon another example here in the form of Mass Effect. Pretty well thought out and executed game, buggy in some places but where my beef with the game resides is from a situation that occurs frequently when I'm playing. So I run into a room full of bad guys, music ups its pace quite well to make things more exciting but .. oh .. I've killed the guys already and suddenly the music returns to its previous state. No warning, no fill, just instant duuun and calm again. It wasn't even on the beat.
Again I don't know if this is me being picky (which I know I am) but that just seemed to ruin it for me. This whole project is to be on ways of getting round the "mass effect" that games have on music. How ways can be developed to allow it to change as often and as subtly as the film composers have so masterfully succeeded in.
Hope that clears some stuff up :D
"What is adaptive music?"
This article by Andrew Clark goes into some detail about how to define adaptive music. However, to effectively grasp this I am going to take a stab at how I perceive the concept.
The interest in computer game audio for me began with one of those early classics (early for me anyway :p) Final Fantasy 7. I loved that game. From the story, to the graphics, to the music, the whole experience drew me in to the make believe world.
When I first bought the game I had been spending time at my grans for a holiday. Ten years old and completely oblivious to pretty much everything. So I get the game, get into the living room where my Playstation was set up and start playing. Needless to say I played that game non-stop throughout that entire holiday and loved every minute of it. My parents however were not as enthusiastic. The main reason - "that awfully repetitive music".
Being only ten I thought it was just "the oldies" showing their age. How could the music in this awesome game be repetitive? I didn't notice anything, it was great! I did however go back and have a play of FF7 a year or 2 ago and damn... that's some repetitive music. Now that my ear has had another ten years to develop I begin to notice what my parents were on about. Sure the music changed when you had a battle, went to a different town or one of the major characters dies ( everyone must have played it by now ) but the music didn't just change, it was more a case of one track ending and another track beginning. This exact situation can be replicated quite easily by loading up your favourite mp3 player and occasionally skipping tracks. Certainly not adaptive music.
So far I took a dig at a game which is over ten years old - ten has seemed to crop up quite a bit in this post - and admittedly games that take the same approach are few and far between when "next-gen" is considered. That's not to say there are games out there that don't do it. Titles like Tony Hawks, Pure and Madden are exempt from this discussion as their music is generally comprised of a playlist containing popular, chart type singles. The games I'm targeting are the types of game that when you play it you're drawn in by either story or a specific game mechanic. Examples of these games are Gears of War, Dawn of War and God of War ( is it just me, or do gamers like war? ).
So, what is adaptive music? What I believe adaptive music to be is a form of playing music in a game that is as dynamic as say, an AI controlled bot in Counterstrike... or the procedural animation used in The Force Unleashed. The music is so unpredictable that even the composer is not too sure what the music will do next, but you can be sure that whatever it does will make sense and not be jarring to fellow audiophiles like myself.
The only effective example of this must be taken not from games, but from film. A similar media but definitely much more predictable and easily demonstrated as far as score is concerned. For this we turn to our pal Indy. Notice the subtle changes in the music to reflect the on screen action. Excellent examples of this are when he takes his gun out to find that it's empty, when "dad" gets the idea to chase the birds and when the focus rests on Indy at the end of the clip the main theme is effortlessly slipped in. This is the kind of subtlety that is missing in modern day computer games as far as in game music is concerned.
That's not to say that games are not trying to be more like films in terms of score, and call me stubborn but I don't think they are doing the beautiful music composed for them justice. I'll quickly draw upon another example here in the form of Mass Effect. Pretty well thought out and executed game, buggy in some places but where my beef with the game resides is from a situation that occurs frequently when I'm playing. So I run into a room full of bad guys, music ups its pace quite well to make things more exciting but .. oh .. I've killed the guys already and suddenly the music returns to its previous state. No warning, no fill, just instant duuun and calm again. It wasn't even on the beat.
Again I don't know if this is me being picky (which I know I am) but that just seemed to ruin it for me. This whole project is to be on ways of getting round the "mass effect" that games have on music. How ways can be developed to allow it to change as often and as subtly as the film composers have so masterfully succeeded in.
Hope that clears some stuff up :D
Tuesday, 30 September 2008
I'd Be Honoured!
Now, down to business.
The topic in which I have chosen to study as part of my honours project is one of the audio variety. Adaptive music in computer games. Even in recent "next-gen" games I still find myself being presented with jarring transitions between, say, the generic "battle theme" switching back to the "overworld theme". This example, albeit now a rare occurrence is something that I feel is unacceptable considering modern processing capabilities.
Without delving too much into the specifics - gotta keep something for the dissertation :P - here is a list of issues surrounding this topic that I intend to investigate as I described in "project worksheet #1" :
These issues have not been set in stone as I'm still in the early stage of development and I want to obtain as much feedback as possible. Already I have been given several suggestions on improving the quality of this project, one of which being to maybe look into how adaptive music affects the content creator. This presents the question "are current audio generation tools up to the task?" and can they cope with the demand put upon the composer. This is certainly something I would want to pursue as it takes the findings and applys them to real life situations making this project more relevant to today.
To summarise - Gee, blogging sure is hard work - a week in to the project and already I've been presented with different ways to approach this topic. I only hope that when the time comes I'll be confident enough to say "this is what my project is about" and hopfully have enough time left to produce something that I am proud of.
The topic in which I have chosen to study as part of my honours project is one of the audio variety. Adaptive music in computer games. Even in recent "next-gen" games I still find myself being presented with jarring transitions between, say, the generic "battle theme" switching back to the "overworld theme". This example, albeit now a rare occurrence is something that I feel is unacceptable considering modern processing capabilities.
Without delving too much into the specifics - gotta keep something for the dissertation :P - here is a list of issues surrounding this topic that I intend to investigate as I described in "project worksheet #1" :
Why should music need to adapt in a game environment?
Difficulties in creating seamless transitions and passages.
Technical/artistic/cost constraints in producing adaptable game scores.
How music in film and television fits the on screen context in ways that games currently do not.
How can various algorithms be applied to enable musical to dynamically tailor itself to the current situation? (e.g. Beat detection, time stretching etc.)
How adaptive music has previously been used in games and how modern technology can improve on these initial developments.
These issues have not been set in stone as I'm still in the early stage of development and I want to obtain as much feedback as possible. Already I have been given several suggestions on improving the quality of this project, one of which being to maybe look into how adaptive music affects the content creator. This presents the question "are current audio generation tools up to the task?" and can they cope with the demand put upon the composer. This is certainly something I would want to pursue as it takes the findings and applys them to real life situations making this project more relevant to today.
To summarise - Gee, blogging sure is hard work - a week in to the project and already I've been presented with different ways to approach this topic. I only hope that when the time comes I'll be confident enough to say "this is what my project is about" and hopfully have enough time left to produce something that I am proud of.
Monday, 29 September 2008
Pop Goes The Weasle
First blog of the ... well... ever. Hopfully subsequent blogs will have more substance but this is just a test. Look forward to future posts about my upcoming honours project.
Hello everyone in blog land! ( could that translate to Bland? Only time will tell ... )
Hello everyone in blog land! ( could that translate to Bland? Only time will tell ... )
Subscribe to:
Posts (Atom)
