db&w documentation wiki

finest software | finest docs

Site Tools


tools:audiofrequency

Audio Frequency Node

This node can load audio files (WAV, MP3, FLAC), performs a FFT 1) analysis on them and then provides the amplitudes at specific frequencies as nodal outputs.

It can be used to drive deformations, animation or shading from audio files.

For the simplest use, load an audio file, press OK in the settings window and connect any of the Bass → High outputs.

You can select frequencies or a range of frequencies couple with a variety of settings to fine tune the output for more specific use.

The sample content includes three sample scenes as well as audio files that show how the Audio Frequency node and channel modifier can be used.

Technical Background

Here is what happens with the audio file.

First it is loaded into memory. Then the FFT runs across it, using the FFT size.

This results in a linear spectrum distribution of FFT / 2 samples per frame for the whole length of the audio.

The problem is that a linear distribution isn't that useful. Humans tend to resolve better at lower frequencies and worse at higher frequencies. Looking at the spectrum of most audio such as music or speech, the useful frequencies tend to be below a few thousand Hz.

This is why the plugin creates frequency bins based on Octaves 2). These are in turn subdivided based on the Values per Octave setting.

This leads to a much more natural, and useful, distribution of frequencies in the spectrum.

Once the frequencies have been binned per frame and stored in a cache, the result of the FFT is deleted. The audio file is ready to be used.

When either of the plugins are used in Modeler, a fps of 30 is assumed since the scene fps is not available. This may affect loading a motion path saved in Layout to be used in Modeler as well.

The fps is used to determined the number of spectrum samples stored in memory per second (one set of samples per frame). If the fps in your scene is different, the result in Modeler will be different as well (especially if interpolate is off).

In that case, baking in Layout is the only way to get a 100% accurate representation.

Controls

The top preview will display the audio spectrum for the current frame. Low frequencies are on the left, high frequencies on the right.

The status below that shows some information about the audio file that's being used, such as the format, length in time and frames, the amount of channels as well as the sampling frequency.

Audio File

This allows you to load a new audio file to use or select one that has been loaded already.

Please note that the settings for an audio file are shared amongst all nodes since all values are pre-computed and cached for performance reasons.

When loading a new audio file, the settings will open after the file has been loaded and before the FFT is being computed.

Use In Layout

This will load the currently selected audio file as the File used in Layout, allowing for playback of the audio file.

This will not work for some types of audio files, Layout seems to only support WAV and MP3 at 16-bit. It will also not match the audio offset if changed in the plugin, as there is no SDK access to that.

Settings...

This opens the settings for the currently selected audio file. These settings are set per audio file, regardless of where it is used in the current scene.

Frequency Mode

This determines which amplitudes are extracted from the audio spectrum.

  • Single This extracts the amplitude at a single frequency.
  • Range (Average) This extracts the average amplitude within the range from Frequency to End Frequency.
  • Range (Peak) This extracts the peak amplitude within the range from Frequency to End Frequency.
  • Range (Smooth) This extracts the average amplitude within the range from Frequency to End Frequency but smoothed with a curve that emphasises the frequencies in the middle of the range.

When a frequency Range is selected, either the average, peak or smoothed average of the frequencies in the range will be computed.

Frequency

The Frequency in Hz that should be extracted. The available frequencies are in the range of 0 (or thereabouts) to half the sampling frequency 3).

End Frequency

If the Frequency mode is set to Range, this setting will control the end of the frequency range. The start is determined by the Frequency setting.

Max. Level (dB) / Min. Level (dB)

This limits the amplitude to be used. Amplitudes between those two levels will be scaled to a 0.0 → 1.0 range.

To increase the amplitude of low volume frequencies, lower the Max. Level.

To remove noisier, low amplitude parts, increase the Min. Level.

Select...

This opens the Spectrum Picker in a separate window to allow for the graphical selection of a frequency or frequency range.

Interpolate

This interpolates the values in the bins depending on the frequency, smoothing out the result.

Time Blending

This setting will average the amplitude around the current frame, making changes to it less sudden and extreme.

  • Off To time blending will take place
  • Average The amplitude will be the average over a range of frames
  • Peak The amplitude will be the highest over a range of frames
  • Smooth The amplitude will be the average over a range of frames with an emphasis on the amplitudes closer to the current frame

Frames +/-

If Time Blending is anything except for off, this allows you to changed the number of frames around the current frame to average the amplitude.

Time Offset (s)

This allows you to offset the time where the audio file is sampled.

Please note, if you envelope this control, it will not be used as an offset, but as the absolute time.

Invert

Inverts the output Amplitude, before the Boost is applied.

Boost

Boosts the output amplitude. This basically multiplies it by the boost value.

Node Inputs

Frequency

This is the frequency for which to get the amplitude. This is in Hz - so a value of 440.0 will extract the amplitude at 440 Hz.

Frequency (norm)

This is the frequency for which to get the amplitude but normalised from 0.0 to 1.0. 0.0 will be 0Hz, 1.0 will be half the sampling rate of the audio file (i.e. 22.05KHz for a 44,100Hz audio file).

Spectrum (norm)

Similar to Frequency (norm), this input actually uses the Octave bins as opposed to the frequencies. This will lead to a much more pleasant distribution if it's used to colour items based on an index or displace vertices depending on their position.

End Frequency

If the Frequency mode is set to Range, this setting will control the end of the frequency range. The start is determined by the Frequency setting.

Minimum / Maximum dB

This limits the amplitude to be used. Amplitudes between those two levels will be scaled to a 0.0 → 1.0 range.

To increase the amplitude of low volume frequencies, lower the Max. Level.

To remove noisier, low amplitude parts, increase the Min. Level.

Time

The absolute time, in seconds, where the audio file should be evaluated.

Time Offset

An offset, in seconds, added to the evaluation time.

Boost

Boosts the output amplitude. This basically multiplies it by the boost value.

Node Outputs

The node provides a set of default outputs for the summed audio channels as well as access to every individual channel in the audio file.

Waveform

The original audio waveform for all channels, from -1.0 to 1.0

Amplitude

The Amplitude at the selected frequency / range for all channels. Usually from 0.0 → 1.0 but may be boosted.

Bass -> High

These are preset outputs for simple use of the node. They correspond to the following frequency ranges:

Bass 20Hz - 140Hz
Mid-Bass 140Hz - 400Hz
Mid 400Hz - 2.6kHz
Upper-Mid 2.6kHz - 5.2kHz
High 5.2kz - 24Hz

These are affected by the Interpolate as well as the Time Offset, Invert and Boost settings as well as the Max/Min dB Levels.

Waveform_*

The original audio waveform for a single channel, from -1.0 to 1.0

Amplitude_*

The Amplitude at the selected frequency / range for a single channel. Usually from 0.0 → 1.0 but may be boosted.


Settings

The settings are per audio file and shared by all plugins that use it across a scene. The previews displays the whole waveform and you will find some information about the loaded file underneath it.

File

This control is available when changing the settings of a loaded audio file. This allows for replacing it with another file. Please note that this will replace the audio file everywhere it is used in the current scene.

Use in Layout

This will load the currently selected audio file as the File used in Layout, allowing for playback of the audio file.

This will not work for some types of audio files, Layout seems to only support WAV and MP3 at 16-bit.

FFT Size

The FFT size determines the amount of samples processed in on go using the FFT. The following options are available:

  • 1024
  • 2048
  • 4096
  • 8192

What do they affect?

  1. The larger the FFT, the smaller the steps between the individual frequencies that are detected.
  2. The larger the FFT, the lower the frequencies that can be detected.
  3. A smaller FFT will be quicker to detect volume changes.

In general, the default of 2048 should be perfectly fine for most cases.

Changing this setting will cause the FFT to re-compute automatically if needed when exiting this panel.

Values per Octave

This is the number of steps per Octave that are used when binning the result of the FFT. The more steps there are, the higher the resolution of the spectrum. The default is a good compromise, more is rarely needed and you should be able to get by with less.

Peak Mode

  • Average
  • Maximum

This determines how the amplitudes from the FFT are treated when they are binned into Octaves. Usually the amplitudes of multiple frequencies will end up in a bin. This setting controls if the average amplitude or the maximum amplitude will be stored.

Floor Value (-dB)

This is the lowest amplitude that will be stored, anything below that is cut off. The default, -96dB, is the signal to noise ratio for 16-bit audio.

Usually there is no reason to change this, especially since the volume range can be limited using the Minimum / Maximum dB controls later on.

Recompute

Recomputes the FFT for the current audio file. This is also required if the Frames per Second is changed for the current scene.

Recompute All

Recomputes the FFT for all audio files. This is required if the Frames per Second is changed for the current scene.


Spectrum Picker

The Spectrum Picker allows for the graphical selection of a frequency or a range of frequencies. It can be launched from any of the plugins and is non-modal 4).

The preview shows the frequency distribution at the current frame as well as the audio samples around the current frame. This preview is live and will update if you scrub or play back the scene.

Preview

Depending on the Frequency Mode, you can either pick a frequency by clicking on the spectrum or define a range by dragging the left or right edge of the selection.

The top and bottom edges can be dragged to change the Max./Min. Level (dB) settings.

A stippled crosshair is drawn under the mouse cursor, the corresponding frequency and amplitude are displayed in the status bar at the bottom left of the panel.

Controls

These mirror the controls in the plugin the picker was called from.


Audio Frequency Channel Modifier

The Audio Frequency channel modifier shares most controls with the node. We'll just document the additional controls here:

Output

This controls the value that's returned by the channel modifier, these correspond to the node outputs:

  • Waveform The audio waveform, in the range from -1 .. 1.0
  • Amplitude The amplitude for the selected frequency or frequency range
  • Bass The amplitude for the range from 20Hz-140Hz
  • Mid-Bass The amplitude for the range from 140Hz-400Hz
  • Mid The amplitude for the range from 400Hz-2.6kHz
  • Upper Mid The amplitude for the range from 2.6kHz-5.2kHz
  • High The amplitude for the range from 5.2kHz-24kHz

Audio Channel

This allows you to select an specific audio channel or the master channel (which is the average of all channels in the audio file).

Channel Mode

This determines how the output of the channel modifier affects the envelope, taking the original envelope value into account:

  • Replace Replaces the envelope by the output of the modifier
  • Add Adds the output of the modifier to the envelope value
  • Multiply Multiplies the output of the modifier by the envelope value
  • Channel - Audio Subtracts the modifier value from the envelope value
  • Audio - Channel Subtracts the envelope value from the channel value

This is the plugin of the month for April/May 2020, made possible by the finest patrons on Patreon.

1)
Fast Fourier Transformation. In this case, Fourier analysis converts a signal from time to a representation in the frequency domain
2)
In music, an octave or perfect octave is the interval between one musical pitch and another with double its frequency.
3)
The Nyquist limit.
4)
Which basically means it can stay open
tools/audiofrequency.txt · Last modified: 2020/09/11 11:42 by lightwolf