Reply
Reply
Edit
Sep 3
Pamela fox, Google I/O 2010 Bot and 249 others:

      Live Wave: Advanced Android audio techniques


      Thursday May 20, at 2:15pm-3:15pm, Room #6 (More Info)

      #android9 | Tweet This | Post to Google Buzz


      Attendees:

      Google I/O Session Attendees

      Questions:

      Google Wave Moderator


      Live Notes: Advanced Android audio techniques

      A designated Googler will be taking notes during this session, but feel free to join in below!


      Overview:

      Raw PCM audio API

      Streaming

      Static mode

      Set callbacks to refill buffer

      Retrieve play position


      Here's the java interface for using AudioTrack...


      public class AudioTrackSample implements Runnable {

      static private final int mBufferSize = 8000;

      private AudioTrack mTrack;

      private short mBuffer[];

      private short mSample;


      class AudioTrackSample {

      mBuffer = new short[mBufferSize];

      mTrack = new AudioTrack(STREAM_MUSIC, 44100, CHANNEL_OUT_MONO,

      ENCODING_PCM_16BIT, mBufferSize * 2, MODE_STREAM);

      mSample = 0;

      }


      public void run() {

      mTrack.play();

      while(1) {

      // fill the buffer

      generateTone(mBuffer, mBufferSize);

      mTrack.write(mBuffer, 0, mBufferSize);

      }

      }


      public void generateTone(short [] data, int size) {

      for (int i = 0; i < size; i++) {

      pData[i] = mSample;

      mSample += 600; // ~400 Hz sawtooth

      }

      }

      }


      That's the basic Java interface.

      IF you want to use native. Download the NDK in addition to the SDK.

      Build a java app and then create a native library, using some sample code from the NDK. The library has implementations of java methods that call native code, plus the code that links java to native.


      Here's some Java code that links to the native code:


      public class JNISample implements Runnable {

      static private final int mBufferSize = 8000;

      private AudioTrack mTrack;

      private short mBuffer[] = new short[mBufferSize];

      private int mSample;

      class JNISample {

      mBuffer = new short[bufferSize];

      mTrack = new AudioTrack(STREAM_MUSIC, 44100, CHANNEL_OUT_MONO,

      ENCODING_PCM_16BIT, mBufferSize * 2, MODE_STREAM);

      mSample = 0;

      }


      public void run() {

      mTrack.play();

      while(1) {

      // fill the buffer

      generateTone(mBuffer, mBufferSize);

      mTrack.write(mBuffer, 0, mBufferSize);

      }

      }


      static {

      System.loadLibrary(generate_tone);

      }


      public native int generateTone(short [] data, int size);

      }


      NOw here's some actual naive code in C++. This needs some matching methods for the java to native linker.


      jint Java_com_example_jnisample_JNISample_generateTone(

      JNIEnv *env, jobject thiz, jshortArray data, jint size)

      {

      jbyte* pData = NULL;

      if (!data || !size) {

      return ERR_NO_DATA;

      }


      // convert java short array to C pointer, pinning the array in memory

      pData = (short*) env->GetPrimitiveArrayCritical(data, NULL);


      if (!pData) {

      return ERR_NO_DATA;

      }


      // fill buffer with 16-bit PCM audio

      short sample = 0;

      for (int i = 0; i < size; i++) {

      pData[i] = sample;

      sample += 600; // ~400 Hz tone

      }


      // unpin the array

      env->ReleasePrimitiveArrayCritical(data, pData, 0);

      return SUCCESS;

      }


      Here's how to access your objects from the native code:


      static jobject mSampleField = 0;


      jint Java_com_example_jnisample_JNISample_generateTone(

      JNIEnv *env, jobject thiz, jshortArray data, jint size)


      {

      if (mSampleField == 0) {

      jclass clazz = env->FindClass(com.example.jnisample.JNISample);


      if (clazz != 0) {

      mSampleField = env->getFieldID(clazz, mSample, S);

      }


      if (mSampleField == 0) {

      return FATAL_ERROR; // fatal error;

      }

      }


      // ... code to pin array and return pointer goes here

      short sample = env->GetShortField(thiz, mSampleField);

      for (int i = 0; i < size; i++) {

      pData[i] = sample;

      sample += 600; // ~400 Hz tone

      }


      env->SetShortField(thiz, mSampleField, sample);

      // ... code to unpin array goes here

      return SUCCESS;

      }



      Some tips:

      - always call release() on media objects. Otherwise you have to wait for GC to clean up. As long as your holding it, there are resources in media being consumed. This can consume a lot of battery.

      - If you have a hardware codec, be sure to power down unneeded hardware

      - Allow other apps can access hardware resources

      - Call release() from onPause() and not just in onDestroy()


      New in Android 2.2:

      - new Audio focus APIs

      - SoundPool improvements

      - route audio to accomodate things like voicemail and BT SCO



      About Audio Focus:


      Applications need to cooperate in terms of how audio is used by multiple applications.


      We want apps to have control of their audio. So we need to give them enough info to make the right choice. So... apps can receive events about what's happening in audio focus and apps can request audio focus when something is happening that needs audio. Audo focus requests can be permanent or transient (something brief, such as a sound notification) and the app being interupted is notified. Apps requesting focus can also include a ducking hint so that other apps can simply lower their volume (eg. when a navigation directive comes in over a song). When done, an app abandons audio focus.


      Transport controls follow the same types of rules. So an app can take control of the headset controls.


      Helper classes allow an app to be backward compatible while taking advantage of these new features.


      Here is where we register our listener and handle state changes. Depending on how your application works, you may need to do things a little differently. We assume here that the start and stop code is centralized and there are no threading issues.


      public void onCreate() {

      super.onCreate();

      mAudioManager = (AudioManager) getSystemService(Context.AUDIO_SERVICE);

      setVolumeControlStream(AudioManager.STREAM_MUSIC);

      ...

      }


      public void onDestroy() {

      mAudioManager.abandonAudioFocus(mListener);

      ...

      }


      public void play() {

      mAudioManager.requestAudioFocus(mListener, AudioManager.STREAM_MUSIC,

      AudioManager.AUDIOFOCUS_GAIN);

      start();

      mPlaying = true;

      }


      private void onStopped() {

      mPlaying = false;

      mAudioManager.abandonAudioFocus(mListener);

      }


      And here's the code that handles audio focus change. In the simple case, we can treat all types of focus change the same way...


      private onAudioFocusChangeListener mListener = new OnAudioFocusListener() {


      public void onAudioFocusChange(int focusChange) {


      switch (focusChange) {

      case AudioManager.AUDIOFOCUS_LOSS:

      mRestart = true;

      if (mPlaying) {

      pause();

      }

      break;

      case AudioManager.AUDIOFOCUS_TRANSIENT:

      case AudioManager.AUDIOFOCUS_TRANSIENT_CAN_DUCK:

      if (mPlaying) {

      mRestart = true;

      pause();

      }

      break;

      case AudioManager.AUDIOFOCUS_GAIN:


      if (!mPlaying && mRestart) {

      mFocusLost = false;

      resume();

      }

      break;

      }

      }

      }


      Here's how you can handle ducking:


      public void onAudioFocusChange(int focusChange) {


      switch (focusChange) {

      case AudioManager.AUDIOFOCUS_LOSS:

      case AudioManager.AUDIOFOCUS_TRANSIENT:

      if (mPlaying) {

      mFocusLost = true;

      pause();

      }

      break;

      case AudioManager.AUDIOFOCUS_TRANSIENT_CAN_DUCK:

      if (mPlaying) {

      mDucking = true;

      mOldVolume = mVolume;

      setVolume(mVolume * 0.125);

      }

      break;

      case AudioManager.AUDIOFOCUS_GAIN:

      if (!mPlaying && mFocusLost) {

      mFocusLost = false;

      resume();

      } else if (mDucking) {

      mDucking = false;

      setVolume(mOldVolume);

      }

      break;

      }

      }


      }


      If your app handles music and spoken content, you probably want to change the behavior based on the content type. Generally, you dont want spoken content to be ducked.



      About SoundPool improvements...


      - There's a new callback when a sound is loaded. Helps because sounds are loaded asynchronously.

      - autoPause() pauses all active tracks

      - autoResume() resumes all previously active tracks



      Audio Routing improvements:

      - STREAM_VOICE_CALL to route audio to earpiece by default

      - setSpeakerOn() to route to speaker

      - startBluetoothSco() to route BT SCO



      Camera improvements...


      - new preview API. can help avoid GC. You can preallocate your buffers. when full, it's returned to you.

      - Compress YUV to JPEG. Improves wifi performance by about 10fps

      - "no thumbnail" mode for JPEG encoders

      - camera reports FOV and focal length parameters. Good for AR applications so you know what the field is that you're seeing.

      - exposure control settings (+1, -1)

      - new portrait mode support for camera



      Roadmap...


      - OpenSL native API to provide audio track stuff in native code

      - OpenAL support. Has a shim over OpenSL. should help port games.

      - Add autio effects processing

      - Expose more low-level APIs.. a Java api that lets you build the player graph in your application. If you have a streaming support that we don't support, previously there hasn't been access to codecs.

      - WebM support. VP8 + Vorbis

      - FLAC decoder

      - AAC-LC encoder for device that don't have hardware support

      - AMR-WB encoder

      (encoders will be open sourced)




      QA:


      Q: When should i call release()? because i don't know when the user is done.

      A: If the user is just pausing, make an intelligent choice. You should save the current play position and then tear-down when the user doesn't come back. Then restore the media at the saved position when they come back.


      Q: Is there a conflict between recording and playback?

      A: That should work. But you might get leakage from speaker back to the mic. We don't have a cancellation for that audio.


      Q: Any plans for bookmarking a position in a stream? to know latency through the playback.

      A: There's no provision for that right now, but there should be in the future.


      Q: Do you support track-level input control? separate apps with different controls.

      A: There's an ability to set the level for each Audio Track.


      Q: How can i access DRM support?

      A: my advice is to stay away from it. We did something early on, but it's now well integrated. We are working on some DRM integration for the next release. The stuff there now is not well supported.


      Q: what are stagefright highlights?

      A: stagefright was the vehicle to support Flash so that we can expose the stack. We realized if we had a stream extractor and so on, and we ended up with a new framework. In the long run, we're moving away from OpenCore. For now, it's mixed and there's support for both. OpenCOre is used for RTSP and authoring. Everything else is Stagefright.


      Q: What about multiple channels?

      A: OpenSLES has answers to that. We're still thinking about how to handle multiple channels like 5.1. We're working on it and will merge with Google TV, which is slightly ahead of us for that.