diff options
Diffstat (limited to 'native_client_sdk/doc_generated/devguide/coding/audio.html')
| -rw-r--r-- | native_client_sdk/doc_generated/devguide/coding/audio.html | 352 |
1 files changed, 352 insertions, 0 deletions
diff --git a/native_client_sdk/doc_generated/devguide/coding/audio.html b/native_client_sdk/doc_generated/devguide/coding/audio.html new file mode 100644 index 0000000..a4b65bb --- /dev/null +++ b/native_client_sdk/doc_generated/devguide/coding/audio.html @@ -0,0 +1,352 @@ +{{+bindTo:partials.standard_nacl_article}} + +<section id="audio"> +<span id="devguide-coding-audio"></span><h1 id="audio"><span id="devguide-coding-audio"></span>Audio</h1> +<div class="contents local topic" id="contents"> +<ul class="small-gap"> +<li><a class="reference internal" href="#reference-information" id="id1">Reference information</a></li> +<li><a class="reference internal" href="#about-the-pepper-audio-api" id="id2">About the Pepper audio API</a></li> +<li><a class="reference internal" href="#digital-audio-concepts" id="id3">Digital audio concepts</a></li> +<li><a class="reference internal" href="#setting-up-the-module" id="id4">Setting up the module</a></li> +<li><p class="first"><a class="reference internal" href="#creating-an-audio-configuration-resource" id="id5">Creating an audio configuration resource</a></p> +<ul class="small-gap"> +<li><a class="reference internal" href="#resources" id="id6">Resources</a></li> +<li><a class="reference internal" href="#sample-frame-count" id="id7">Sample frame count</a></li> +<li><a class="reference internal" href="#supported-audio-configurations" id="id8">Supported audio configurations</a></li> +</ul> +</li> +<li><a class="reference internal" href="#creating-an-audio-resource" id="id9">Creating an audio resource</a></li> +<li><p class="first"><a class="reference internal" href="#implementing-a-callback-function" id="id10">Implementing a callback function</a></p> +<ul class="small-gap"> +<li><a class="reference internal" href="#application-threads-and-real-time-requirements" id="id11">Application threads and real-time requirements</a></li> +</ul> +</li> +<li><a class="reference internal" href="#starting-and-stopping-playback" id="id12">Starting and stopping playback</a></li> +</ul> +</div> +<p>This chapter describes how to use the Pepper audio API to play an audio +stream. The Pepper audio API provides a low-level means of playing a stream of +audio samples generated by a Native Client module. The API generally works as +follows: A Native Client module creates an audio resource that represents an +audio stream, and tells the browser to start or stop playing the audio +resource. The browser calls a function in the Native Client module to fill a +buffer with audio samples every time it needs data to play from the audio +stream.</p> +<p>The code examples in this chapter describe a simple Native Client module that +generates audio samples using a sine wave with a frequency of 440 Hz. The module +starts playing the audio samples as soon as it is loaded into the browser. For a +slightly more sophisticated example, see the <code>audio</code> example (source code in +the SDK directory <code>examples/api/audio</code>), which lets users specify a frequency +for the sine wave and click buttons to start and stop audio playback.</p> +<section id="reference-information"> +<h2 id="reference-information">Reference information</h2> +<p>For reference information related to the Pepper audio API, see the following +documentation:</p> +<ul class="small-gap"> +<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/classpp_1_1_audio_config">pp::AudioConfig class</a></li> +<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/classpp_1_1_audio">pp::Audio class</a></li> +<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/audio__config_8h">audio_config.h</a></li> +<li><a class="reference external" href="https://developers.google.com/native-client/peppercpp/audio_8h">audio.h</a></li> +<li><a class="reference external" href="https://developers.google.com/native-client/pepperc/group___enums.html#gaee750c350655f2fb0fe04c04029e0ff8">PP_AudioSampleRate</a></li> +</ul> +</section><section id="about-the-pepper-audio-api"> +<h2 id="about-the-pepper-audio-api">About the Pepper audio API</h2> +<p>The Pepper audio API lets Native Client modules play audio streams in a +browser. To play an audio stream, a module generates audio samples and writes +them into a buffer. The browser reads the audio samples from the buffer and +plays them using an audio device on the client computer.</p> +<img alt="/native-client/images/pepper-audio-buffer.png" src="/native-client/images/pepper-audio-buffer.png" /> +<p>This mechanism is simple but low-level. If you want to play plain sound files in +a web application, you may want to consider higher-level alternatives such as +using the HTML <code><audio></code> tag, JavaScript, or the new <a class="reference external" href="http://chromium.googlecode.com/svn/trunk/samples/audio/index.html">Web Audio API</a>.</p> +<p>The Pepper audio API is a good option for playing audio data if you want to do +audio processing in your web application. You might use the audio API, for +example, if you want to apply audio effects to sounds, synthesize your own +sounds, or do any other type of CPU-intensive processing of audio +samples. Another likely use case is gaming applications: you might use a gaming +library to process audio data, and then simply use the audio API to output the +processed data.</p> +<p>The Pepper audio API is straightforward to use:</p> +<ol class="arabic simple"> +<li>Your module creates an audio configuration resource and an audio resource.</li> +<li>Your module implements a callback function that fills an audio buffer with +data.</li> +<li>Your module invokes the StartPlayback and StopPlayback methods of the audio +resource (e.g., when certain events occur).</li> +<li>The browser invokes your callback function whenever it needs audio data to +play. Your callback function can generate the audio data in a number of +ways—e.g., it can generate new data, or it can copy pre-mixed data into the +audio buffer.</li> +</ol> +<p>This basic interaction is illustrated below, and described in detail in the +sections that follow.</p> +<img alt="/native-client/images/pepper-audio-api.png" src="/native-client/images/pepper-audio-api.png" /> +</section><section id="digital-audio-concepts"> +<h2 id="digital-audio-concepts">Digital audio concepts</h2> +<p>Before you use the Pepper audio API, it’s helpful to understand a few concepts +that are fundamental to how digital audio is recorded and played back:</p> +<dl class="docutils"> +<dt>sample rate</dt> +<dd>the number of times an input sound source is sampled per second; +correspondingly, the number of samples that are played back per second</dd> +<dt>bit depth</dt> +<dd>the number of bits used to represent a sample</dd> +<dt>channels</dt> +<dd>the number of input sources recorded in each sampling interval; +correspondingly, the number of outputs that are played back simultaneously +(typically using different speakers)</dd> +</dl> +<p>The higher the sample rate and bit depth used to record a sound wave, the more +accurately the sound wave can be reproduced, since it will have been sampled +more frequently and stored using a higher level of quantization. Common sampling +rates include 44,100 Hz (44,100 samples/second, the sample rate used on CDs), +and 48,000 Hz (the sample rate used on DVDs and Digital Audio Tapes). A common +bit depth is 16 bits per sample, and a common number of channels is 2 (left and +right channels for stereo sound).</p> +<p id="pepper-audio-configurations">The Pepper audio API currently lets Native Client modules play audio streams +with the following configurations:</p> +<ul class="small-gap"> +<li><strong>sample rate</strong>: 44,100 Hz or 48,000 Hz</li> +<li><strong>bit depth</strong>: 16</li> +<li><strong>channels</strong>: 2 (stereo)</li> +</ul> +</section><section id="setting-up-the-module"> +<h2 id="setting-up-the-module">Setting up the module</h2> +<p>The code examples below describe a simple Native Client module that generates +audio samples using a sine wave with a frequency of 440 Hz. The module starts +playing the audio samples as soon as it is loaded into the browser.</p> +<p>The Native Client module is set up by implementing subclasses of the +<code>pp::Module</code> and <code>pp::Instance</code> classes, as normal.</p> +<pre class="prettyprint"> +class SineSynthInstance : public pp::Instance { + public: + explicit SineSynthInstance(PP_Instance instance); + virtual ~SineSynthInstance() {} + + // Called by the browser once the NaCl module is loaded and ready to + // initialize. Creates a Pepper audio context and initializes it. Returns + // true on success. Returning false causes the NaCl module to be deleted + // and no other functions to be called. + virtual bool Init(uint32_t argc, const char* argn[], const char* argv[]); + + private: + // Function called by the browser when it needs more audio samples. + static void SineWaveCallback(void* samples, + uint32_t buffer_size, + void* data); + + // Audio resource. + pp::Audio audio_; + + ... + +}; + +class SineSynthModule : public pp::Module { + public: + SineSynthModule() : pp::Module() {} + ~SineSynthModule() {} + + // Create and return a SineSynthInstance object. + virtual pp::Instance* CreateInstance(PP_Instance instance) { + return new SineSynthInstance(instance); + } +}; +</pre> +</section><section id="creating-an-audio-configuration-resource"> +<h2 id="creating-an-audio-configuration-resource">Creating an audio configuration resource</h2> +<section id="resources"> +<h3 id="resources">Resources</h3> +<p>Before the module can play an audio stream, it must create two resources: an +audio configuration resource and an audio resource. Resources are handles to +objects that the browser provides to module instances. An audio resource is an +object that represents the state of an audio stream, including whether the +stream is paused or being played back, and which callback function to invoke +when the samples in the stream’s buffer run out. An audio configuration resource +is an object that stores configuration data for an audio resource, including the +sampling frequency of the audio samples, and the number of samples that the +callback function must provide when the browser invokes it.</p> +</section><section id="sample-frame-count"> +<h3 id="sample-frame-count">Sample frame count</h3> +<p>Prior to creating an audio configuration resource, the module should call +<code>RecommendSampleFrameCount</code> to obtain a <em>sample frame count</em> from the +browser. The sample frame count is the number of samples that the callback +function must provide per channel each time the browser invokes the callback +function. For example, if the sample frame count is 4096 for a stereo audio +stream, the callback function must provide a 8192 samples (4096 for the left +channel and 4096 for the right channel).</p> +<p>The module can request a specific sample frame count, but the browser may return +a different sample frame count depending on the capabilities of the client +device. At present, <code>RecommendSampleFrameCount</code> simply bound-checks the +requested sample frame count (see <code>include/ppapi/c/ppb_audio_config.h</code> for the +minimum and maximum sample frame counts, currently 64 and 32768). In the future, +<code>RecommendSampleFrameCount</code> may perform a more sophisticated calculation, +particularly if there is an intrinsic buffer size for the client device.</p> +<p>Selecting a sample frame count for an audio stream involves a tradeoff between +latency and CPU usage. If you want your module to have short audio latency so +that it can rapidly change what’s playing in the audio stream, you should +request a small sample frame count. That could be useful in gaming applications, +for example, where sounds have to change frequently in response to game +action. However, a small sample frame count results in higher CPU usage, since +the browser must invoke the callback function frequently to refill the audio +buffer. Conversely, a large sample frame count results in higher latency but +lower CPU usage. You should request a large sample frame count if your module +will play long, uninterrupted audio segments.</p> +</section><section id="supported-audio-configurations"> +<h3 id="supported-audio-configurations">Supported audio configurations</h3> +<p>After the module obtains a sample frame count, it can create an audio +configuration resource. Currently the Pepper audio API supports audio streams +with the configuration settings shown <a class="reference internal" href="#pepper-audio-configurations"><em>above</em></a>. +C++ modules can create a configuration resource by instantiating a +<code>pp::AudioConfig</code> object. Check <code>audio_config.h</code> for the latest +configurations that are supported.</p> +<pre class="prettyprint"> +bool SineSynthInstance::Init(uint32_t argc, + const char* argn[], + const char* argv[]) { + + // Ask the browser/device for an appropriate sample frame count size. + sample_frame_count_ = + pp::AudioConfig::RecommendSampleFrameCount(PP_AUDIOSAMPLERATE_44100, + kSampleFrameCount); + + // Create an audio configuration resource. + pp::AudioConfig audio_config = pp::AudioConfig(this, + PP_AUDIOSAMPLERATE_44100, + sample_frame_count_); + + // Create an audio resource. + audio_ = pp::Audio(this, + audio_config, + SineWaveCallback, + this); + + // Start playback when the module instance is initialized. + return audio_.StartPlayback(); +} +</pre> +</section></section><section id="creating-an-audio-resource"> +<h2 id="creating-an-audio-resource">Creating an audio resource</h2> +<p>Once the module has created an audio configuration resource, it can create an +audio resource. To do so, it instantiates a <code>pp::Audio</code> object, passing in a +pointer to the module instance, the audio configuration resource, a callback +function, and a pointer to user data (data that is used in the callback +function). See the example above.</p> +</section><section id="implementing-a-callback-function"> +<h2 id="implementing-a-callback-function">Implementing a callback function</h2> +<p>The browser calls the callback function associated with an audio resource every +time it needs more samples to play. The callback function can generate new +samples (e.g., by applying sound effects), or copy pre-mixed samples into the +audio buffer. The example below generates new samples by computing values of a +sine wave.</p> +<p>The last parameter passed to the callback function is generic user data that the +function can use in processing samples. In the example below, the user data is a +pointer to the module instance, which includes member variables +<code>sample_frame_count_</code> (the sample frame count obtained from the browser) and +<code>theta_</code> (the last angle that was used to compute a sine value in the previous +callback; this lets the function generate a smooth sine wave by starting at that +angle plus a small delta).</p> +<pre class="prettyprint"> +class SineSynthInstance : public pp::Instance { + public: + ... + + private: + static void SineWaveCallback(void* samples, + uint32_t buffer_size, + void* data) { + + // The user data in this example is a pointer to the module instance. + SineSynthInstance* sine_synth_instance = + reinterpret_cast<SineSynthInstance*>(data); + + // Delta by which to increase theta_ for each sample. + const double delta = kTwoPi * kFrequency / PP_AUDIOSAMPLERATE_44100; + // Amount by which to scale up the computed sine value. + const int16_t max_int16 = std::numeric_limits<int16_t>::max(); + + int16_t* buff = reinterpret_cast<int16_t*>(samples); + + // Make sure we can't write outside the buffer. + assert(buffer_size >= (sizeof(*buff) * kChannels * + sine_synth_instance->sample_frame_count_)); + + for (size_t sample_i = 0; + sample_i < sine_synth_instance->sample_frame_count_; + ++sample_i, sine_synth_instance->theta_ += delta) { + + // Keep theta_ from going beyond 2*Pi. + if (sine_synth_instance->theta_ > kTwoPi) { + sine_synth_instance->theta_ -= kTwoPi; + } + + // Compute the sine value for the current theta_, scale it up, + // and write it into the buffer once for each channel. + double sin_value(std::sin(sine_synth_instance->theta_)); + int16_t scaled_value = static_cast<int16_t>(sin_value * max_int16); + for (size_t channel = 0; channel < kChannels; ++channel) { + *buff++ = scaled_value; + } + } + } + + ... +}; +</pre> +<section id="application-threads-and-real-time-requirements"> +<h3 id="application-threads-and-real-time-requirements">Application threads and real-time requirements</h3> +<p>The callback function runs in a background application thread. This allows audio +processing to continue even when the application is busy doing something +else. If the main application thread and the callback thread access the same +data, you may be tempted to use a lock to control access to that data. You +should avoid the use of locks in the callback thread, however, as attempting to +acquire a lock may cause the thread to get swapped out, resulting in audio +dropouts.</p> +<p>In general, you must program the callback thread carefully, as the Pepper audio +API is a very low level API that needs to meet hard real-time requirements. If +the callback thread spends too much time processing, it can easily miss the +real-time deadline, resulting in audio dropouts. One way the callback thread can +miss the deadline is by taking too much time doing computation. Another way the +callback thread can miss the deadline is by executing a function call that swaps +out the callback thread. Unfortunately, such function calls include just about +all C Run-Time (CRT) library calls and Pepper API calls. The callback thread +should therefore avoid calls to malloc, gettimeofday, mutex, condvars, critical +sections, and so forth; any such calls could attempt to take a lock and swap out +the callback thread, which would be disastrous for audio playback. Similarly, +the callback thread should avoid Pepper API calls. Audio dropouts due to thread +swapping can be very rare and very hard to track down and debug—it’s best to +avoid making system/Pepper calls in the first place. In short, the audio +(callback) thread should use “lock-free” techniques and avoid making CRT library +calls.</p> +<p>One other issue to be aware of is that the <code>StartPlayback</code> function (discussed +below) is an asynchronous RPC; i.e., it does not block. That means that the +callback function may not be called immediately after the call to +<code>StartPlayback</code>. If it’s important to synchronize the callback thread with +another thread so that the audio stream starts playing simultaneously with +another action in your application, you must handle such synchronization +manually.</p> +</section></section><section id="starting-and-stopping-playback"> +<h2 id="starting-and-stopping-playback">Starting and stopping playback</h2> +<p>To start and stop audio playback, the module simply reacts to JavaScript +messages.</p> +<pre class="prettyprint"> +const char* const kPlaySoundId = "playSound"; +const char* const kStopSoundId = "stopSound"; + +void SineSynthInstance::HandleMessage(const pp::Var& var_message) { + if (!var_message.is_string()) { + return; + } + std::string message = var_message.AsString(); + if (message == kPlaySoundId) { + audio_.StartPlayback(); + } else if (message == kStopSoundId) { + audio_.StopPlayback(); + } else if (...) { + ... + } +} +</pre> +</section></section> + +{{/partials.standard_nacl_article}} |