SampleTagger, fast indexer for Universal Patch Finder

• Download code - 30.5 KB

Introduction

Universal Patch Finder (UPF) is a freeware designed to index various audio files from multiple directories. Every file can be tagged manually or automatically using plug-ins. The purpose of this article is to demonstrate how the plug-in SampleTagger, part of UPF 1.4, is designed.

Background

If you are not familiar with the UPF SDK, you can read this article.

New way to tag in UPF 1.4

UPF 1.4 has introduced a new method to tag a patch relying on renaming rules declared in the INI file. The plug-in is no longer responsible for determining the nature of a tag (type, domain...). You just have to tag the patch with a name and a value using the method CreateTagWithoutType, and UPF will determine the nature of it regarding the .INI file.

Here the new UPFContext providing CreateTagWithoutType:

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate void CreateTagWithoutTypeDelegate(String name, String value);

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct UPFContext
{
    public CreateTagDelegate CreateTag; // method to create a new tag
    public String LibraryPath;          // directory of the library
    public String PatchPath;            // relative patch of the patch
    public String Fullpath;             // full path of the patch
    public IntPtr Handle;               // low level stream handle
    public FileStream Stream;           // high level stream handle
    public Int32  CRC32;                // checksum of the patch
    public Object Patch;                // internal use only
    
    // method to create a new tag without provinding any type. 
    // UniversalPatchFinder.ini must contain the definition of it
    // (if not, the method does nothing)
    public CreateTagWithoutTypeDelegate CreateTagWithoutType; 
};

About audio file formats

Implement a parser for AIFF, WAV and REX files is a matter of chunks as they are defined in RIFF (Resource Interchange File Format) specification by Microsoft in 1992. Note that only WAV are intended to be RIFF files but, in fact it's nearly the same thing.

• WAV is a file format designed by Microsoft. We will focus only on uncompressed PCM audio files.
• AIFF is a file format designed by Apple.
• RX2 is the latest file format designed by Propellerhead, specializing on audio loops (previous ones are REX and RCY).

Beyond these three file formats, there are some derivatives like:

Format	Based on
Apple loops	AIFF
RCY	AIFF
REX	AIFF
ACID	WAV

Some proprietary and undocumented chunks have been introduced by companies: 

• 'acid', 'trns', 'cate' introduced by Sony for their ACID-WAV.
• 'AFmd' introduced by Apple (?) in WAV. Seems to be the result of a NSKeyedArchiver.
• 'APPL' introduced by Propellerhead for their RCY and REX. 

Custom parser vs. official parsers

REX, RCY, and RX2 files can be parsed by the official Propellerhead SDK, but it requires to load the entire file in memory which is not practical if you intend to index tons of files like us. SampleTagger is 80% faster than the official API. Here is a benchmark I made:

• input: 1579 REX files  (1.67 GB of audio data)
• rex SDK: 118324 sec  
• rex parser: 21556 sec 

Damaged audio files

It's not unusual to encounter damaged audio files, where the size of a chunk is wrong. Most of the time there is one byte '0x00' that should not be here at the end of the data. Certainly because the authoring software have written the final byte of a C String ! So our parser must handle this properly. NAudio for instance, don't like this at all.

Classes hierarchy

BinaryFileReader is responsible to read all chunks and take care of the damaged ones. Endianness is also provided by this abstract class.

• REXReader will parse RX2 files
• AIFFReader parses Apple loops, RCY, REX and regular AIFF files
• WAVReader parses ACID WAV and regular WAV files 

Each parser is intended to populate a class SampleFileInfo in the following way: 

info.AddField(PredefinedPatchAttribute.TEMPO, tempo);
info.AddField(PredefinedPatchAttribute.TIME_SIGNATURE, meter_numerator + "/" + meter_denominator);
info.AddField(PredefinedPatchAttribute.BEATS, number_of_beats);
info.AddField(PredefinedPatchAttribute.BARS, number_of_bars); 

Reversing proprietary chunks

To reverse the format of a proprietary chunk, you need a generator and a decent hexadecimal editor. You change one field in a file, then you compare with the previous version.  Here some generators:

• Apple loop can be created with the Apple Loop Utility: 



• ACID WAV can be created with ACID Pro by Sony. 
• RX2 file can be created with the latest Recycle by Propellerhead 
• RCY and REX files can be created with the deprecated version 1.71 of ReCycle 
• Hexadecimal comparison can be achieved using HxD which is great for this job.

Useful links:

• ACID chunk: https://github.com/michaelwu/libsndfile/blob/master/src/wav.c 
• Apple loop: http://analoq.noisepages.com/2009/12/reverse-engineering-apple-loops/
• REX 2: code from Valek Filippov in re-lab tools

Points of Interest

• AIFF sampling frequency in known to use the infamous 80 bits long double format. Unfortunately .NET does not provide a way to read it, so we use the code provided in this article.
• AIFF and REX files use big endian.
• WAV files use little endian.