For those that follow, the problem was indeed an incorrect declaration of the external methods. Some of the parameters declared as "int" should be "IntPtr" (which is of course different for 32 and 64-bit platforms), so always check the original API documentation and don't just assume code you copy from these pages is correct! These specific methods should be:
[DllImport("avifil32.dll", PreserveSig = true)]
public static extern int AVIFileOpen(ref IntPtr ppfile, string szFile, uint uMode, IntPtr pclsidHandler);
[DllImport("avifil32.dll")]
public static extern int AVIFileGetStream(IntPtr pfile, out IntPtr ppavi, uint fccType, int lParam);