Click here to Skip to main content
Click here to Skip to main content

Implementing a TextReader to extract various files contents using IFilter

, 9 Feb 2011 Eclipse
Rate this:
Please Sign up or sign in to vote.
A solution that can extract various file contents using an IFilter implementation. Special thanks to Eyal Post and his article 'Using IFilter in C# '.

Introduction

The IFilter interface is an important component for the Microsoft Indexing Service. The intent of this project is to provide a solution which is high performing and also has a low memory footprint. This will be accomplished by using a TextReader and manipulating a wrapper of IFilter to interop with managed code.

The article is written to target users who have had development experience with COM in a managed environment, and is aimed at creating a solution that can extract various file contents using an IFilter implementation.

The solution in the source code package contains a test project, which includes a Unit Test and a load test. These were created using a higher edition of Visual Studio. If you cannot open the test project, you can always remove the IFilterTest project.

Background

The IFilter component is an in-process COM server that extracts the text and values for a specific file type. The appropriate IFilter component for a file type is called by the Filtering component.

A customized IFilter component can be developed for almost any selected file type. The standard IFilter components supplied with Indexing Service include the following:

File Name

Description

mimefilt.dll

Filters Multipurpose Internet Mail Extension (MIME) files.

nlhtml.dll

Filters HTML 3.0 or earlier files.

offfilt.dll

Filters Microsoft Office files: Microsoft Word, Microsoft Excel, and Microsoft PowerPoint®.

query.dll

Filters plain text files (default filter) and binary files (null filter).

The IFilter interface scans documents for text and properties (also called attributes). It extracts chunks of text from these documents, filtering out embedded formatting and retaining information about the position of the text. It also extracts chunks of values, which are properties of an entire document or of well-defined parts of a document. IFilter provides the foundation for building higher-level applications such as document indexers and application-independent viewers.

Picture1.png

Main Classes Diagram

IFilter Interface and Mixed Class

The IFilter code:

[ComImport, Guid(Constants.IFilterGUID), 
    InterfaceType(ComInterfaceType.InterfaceIsIUnknown),
    SuppressUnmanagedCodeSecurity, ComVisible(true), AutomationProxy(false)]
public interface IFilter
{
    /// <summary>
    /// The IFilter::Init method initializes a filtering session.
    /// </summary>
    [PreserveSig]
    [MethodImpl(MethodImplOptions.InternalCall,
                MethodCodeType = MethodCodeType.Runtime)]
    IFilterReturnCodes Init(
        //[in] Flag settings from the IFILTER_INIT enumeration for
        // controlling text standardization, property output, embedding
        // scope, and IFilter access patterns. 
        [MarshalAs(UnmanagedType.U4)]IFILTER_INIT grfFlags,
        // [in] The size of the attributes array. When nonzero, cAttributes
        //  takes 
        // precedence over attributes specified in grfFlags. If no
        // attribute flags 
        // are specified and cAttributes is zero, the default is given by
        // the 
        // PSGUID_STORAGE storage property set, which contains the date and
        //  time 
        // of the last write to the file, size, and so on; and by the
        //  PID_STG_CONTENTS 
        // 'contents' property, which maps to the main contents of the
        // file. 
        // For more information about properties and property sets, see
        // Property Sets. 
        uint cAttributes,
        //[in] Array of pointers to FULLPROPSPEC structures for the
        // requested properties. 
        // When cAttributes is nonzero, only the properties in aAttributes
        // are returned. 
        FULLPROPSPEC[] aAttributes,
        // [out] Information about additional properties available to the
        //  caller; from the IFILTER_FLAGS enumeration. 
        out IFILTER_FLAGS pdwFlags);
    /// <summary>
    /// The IFilter::GetChunk method positions the filter at the beginning
    /// of the next chunk, 
    /// or at the first chunk if this is the first call to the GetChunk
    /// method, and returns a description of the current chunk. 
    /// </summary>
    [PreserveSig]
    [MethodImpl(MethodImplOptions.InternalCall,
     MethodCodeType = MethodCodeType.Runtime)]
    IFilterReturnCodes GetChunk(out STAT_CHUNK pStat);
    /// <summary>
    /// The IFilter::GetText method retrieves text (text-type properties)
    /// from the current chunk, 
    /// which must have a CHUNKSTATE enumeration value of CHUNK_TEXT.
    /// </summary>
    [PreserveSig]
    [MethodImpl(MethodImplOptions.InternalCall,
     MethodCodeType = MethodCodeType.Runtime)]
    IFilterReturnCodes GetText(
        // [in/out] On entry, the size of awcBuffer array in wide/Unicode
        // characters. On exit, the number of Unicode characters written to
        // awcBuffer. 
        // Note that this value is not the number of bytes in the buffer. 
        ref uint pcwcBuffer,
        // Text retrieved from the current chunk. Do not terminate the
        // buffer with a character.
        [Out]IntPtr awcBuffer);
    /// <summary>
    /// The IFilter::GetValue method retrieves a value (public
    /// value-type property) from a chunk, 
    /// which must have a CHUNKSTATE enumeration value of CHUNK_VALUE.
    /// </summary>
    [PreserveSig]
    [MethodImpl(MethodImplOptions.InternalCall,
     MethodCodeType = MethodCodeType.Runtime)]
    IFilterReturnCodes GetValue(
        // Allocate the PROPVARIANT structure with CoTaskMemAlloc. Some
        // PROPVARIANT 
        // structures contain pointers, which can be freed by calling the
        // PropVariantClear function. 
        // It is up to the caller of the GetValue method to call the
        //   PropVariantClear method.
        // ref IntPtr ppPropValue
        // [MarshalAs(UnmanagedType.Struct)]
        out PROPVARIANT PropVal);
    /// <summary>
    /// The IFilter::BindRegion method retrieves an interface representing
    /// the specified portion of the object. 
    /// Currently reserved for future use.
    /// </summary>
    [PreserveSig]
    [MethodImpl(MethodImplOptions.InternalCall,
     MethodCodeType = MethodCodeType.Runtime)]
    IFilterReturnCodes BindRegion(ref FILTERREGION origPos,ref Guid riid,
        ref object ppunk);
}

The mixed class:

public class MixedIFilterClass : IFilterClass, IDisposable
{
    public override string TmpFilePath
    {
        get;
        set;
    }
    public override Object InternalObj
    { 
        get;
        set;
    }
    //private MixedIFilterClass()
    //{
    //    InternalPtr = Marshal.GetComInterfaceForObject(this, typeof(IFilter));
    //}
    ~MixedIFilterClass()
    {
        Dispose(false);
    }
    protected virtual void Dispose(bool disposing)
    {
        if(null != InternalObj)
        {
            Marshal.ReleaseComObject(InternalObj);
            InternalObj = null;
        }
        if (null != TmpFilePath)
            try
            {
                File.Delete(TmpFilePath);
                TmpFilePath = null;
            }
            catch { }
        if (disposing)
            GC.SuppressFinalize(this);
    }
    public void Dispose()
    {
        Dispose(true);
    }
}

How it Works

There are two steps needed to show how the process works They are:

  1. Get the current chunk
  2. Call GetText() on the chunk

Step 1: Get the current chunk

If you reach the last chunk, terminate the reading process.

var returnCode = _filter.GetChunk(out chunk);

Step 2: Call GetText() on the chunk

Depending on the state gotten from the GetChunk method, call the GetText method on the text chunk. When reading the end of current chunk flag, repeat step 1.

while (true)
{
    if (remaining <= _topSize)
        return;
    bool useBuffer = !forceDirectlyWrite && remaining < BufferSize;
    var size = BufferSize;
    if (useBuffer)
        size -= _topSize;
    else
    {
        if (remaining < BufferSize)
            size = (uint)remaining;
    }
    if (size < ResBufSize)
        size = ResBufSize;
    var handle = GCHandle.Alloc(useBuffer ? _buffer : array,
        GCHandleType.Pinned);
    var ptr = Marshal.UnsafeAddrOfPinnedArrayElement(
        useBuffer ? _buffer : array, useBuffer ? (int)_topSize : offset);
    IFilterReturnCodes returnCode;
    try
    {
#if DEBUG
        Trace.Write(size);
#endif
        returnCode = _filter.GetText(ref size, ptr);
#if DEBUG
        Trace.WriteLine("->"+size);
#endif
    }
    finally 
    {
        handle.Free();
    }
    if(returnCode != IFilterReturnCodes.FILTER_E_NO_TEXT)
    {
        if (useBuffer)
            _topSize += size;
        else
        {
            offset += (int)size;
            remaining -= (int)size;
        }
        if(_topSize > BufferSize)
        {
            _resTopSize = _topSize - BufferSize;
            _topSize = BufferSize;
        }
    }
    if (returnCode == IFilterReturnCodes.FILTER_S_LAST_TEXT || 
        returnCode == IFilterReturnCodes.FILTER_E_NO_MORE_TEXT ||
        (returnCode == IFilterReturnCodes.FILTER_E_NO_TEXT && size != 0) ||
        (null == FileName && IgnoreError && returnCode == 
        IFilterReturnCodes.E_INVALIDARG))
    {
        _endOfCurrChunk = true;
        if (remaining <= _topSize)
            return;
        break;
    }
    if(returnCode != IFilterReturnCodes.S_OK)
    {
        throw new Exception(
            "a error occur when getting text by current filter",
            new Exception(returnCode.ToString()));
    }
}

Using the Code

The following code uses just a filename:

var fileName = "";
using (var reader = new FilterReader(fileName))
{
    reader.Init();
    //
    // write your code here;
    //
}

This code will specify a file and an extension:

using (var reader = new FilterReader(fileName, ".docx"))
{
    reader.Init();
    //
    // write your code here;
    //
}
using (var reader = new FilterReader(fileName, 0x1000))
{
    reader.Init();
    //
    // write your code here;
    //
}

The code below shows how to pass a byte array into a FilterReader:

byte[] bytes = null;
using (var reader = new FilterReader(bytes, ".docx"))
{
    reader.Init();
    //
    // write your code here;
    //
}
using (var reader = new FilterReader(bytes, ".docx", 0x1000))
{
    reader.Init();
    //
    // write your code here;
    //
}
using (var reader = new FilterReader(bytes))
{
    reader.Init();
    //
    // write your code here;
    //
}

Reference

History

  • 2/9/2010 - Updated download files.
  • 3/03/2009 - v2.0: Replaced Copyright and added OLE FMTIDs, Windows Search Schema, and OLE property interfaces for further needs.
  • 2/05/2009 - v2.0: Reconstructed some phrases (thanks Sean Kenney for reviewing this article).
  • 1/07/2009 - v2.0: Added comments for Adobe's PDF filter and small changes.
  • 12/09/2008 - v2.0 [Stable release].
  • 11/24/2008 - v1.0 [Initial release].

License

This article, along with any associated source code and files, is licensed under The Eclipse Public License 1.0

Share

About the Author

alex_zero
Technical Lead HP
China China
No Biography provided

Comments and Discussions

 
BugException from HRESULT : 0x8004170C PinmemberMember 1096235812-Oct-14 6:11 
SuggestionSupport for PDF file indexing [modified] PinmemberDaniel Kornev4-May-14 15:03 
GeneralRe: Support for PDF file indexing Pinmembervladimir-v7722-Sep-14 0:39 
QuestionURGENT: Unable to extract text from docx, xlsx files PinmemberShiva Wahi16-Oct-13 21:23 
AnswerRe: URGENT: Unable to extract text from docx, xlsx files Pinmembervladimir-v773-Oct-14 3:19 
QuestionUnable to retrieve content from .pdf, .docx, .xlsx... PinmemberRicardoRomão1-Jul-13 8:44 
AnswerRe: Unable to retrieve content from .pdf, .docx, .xlsx... Pinmembersantoshthankachan28-Aug-13 20:54 
QuestionLoading from a Stream? PinmemberMatt Johnson14-Jan-13 4:25 
BugPeek() still doesn't work PinmemberMember 874096323-May-12 5:13 
BugFrequent use of FilterReader() breaks it PinmemberSnæbjørn23-Apr-12 2:59 
QuestionUnable to extract content from rtf file (Rich Text Format) Pinmemberajayk.gupta20-Feb-12 23:21 
AnswerRe: Unable to extract content from rtf file (Rich Text Format) [modified] PinmemberSnæbjørn23-Apr-12 5:16 
GeneralMy vote of 5 Pinmembermanoj kumar choubey9-Feb-12 23:15 
GeneralMy vote of 5 PinmemberJonathan Manley29-Jun-11 10:19 
QuestionReading document properties by using the IFilter PinmemberRamkrishnan Seerangasamy1-Jun-11 1:07 
GeneralMy vote of 5 Pinmemberalessandro7217-Feb-11 0:59 
This solution is clear, complete and structured according to Microsoft GuideLine.
It's an excellent example to learn IFilter object
Generala bug while text extracting from "docx" PinmemberEmre Özgür İnce11-Feb-11 3:40 
GeneralRe: a bug while text extracting from "docx" Pinmemberalex_zero11-Feb-11 19:58 
GeneralRe: a bug while text extracting from "docx" PinmemberEmre Özgür İnce13-Feb-11 21:36 
GeneralRe: a bug while text extracting from "docx" Pinmemberalex_zero14-Feb-11 18:23 
GeneralRe: a bug while text extracting from "docx" PinmemberEmre Özgür İnce15-Feb-11 0:29 
QuestionPeek Fails on some file types - Any Ideas? PinmemberErra erra eric14-Dec-10 12:30 
AnswerRe: Peek Fails on some file types - Any Ideas? Pinmemberalex_zero6-Feb-11 23:50 
GeneralRe: Peek Fails on some file types - Any Ideas? PinmemberErra erra eric8-Mar-11 11:52 
GeneralCreating an instance of the COM component from the IClassFactory failed Pinmemberbelikemike12129-Jul-10 13:55 
GeneralRe: Creating an instance of the COM component from the IClassFactory failed Pinmemberalex_zero15-Aug-10 5:25 
GeneralRe: Creating an instance of the COM component from the IClassFactory failed Pinmemberalex_zero6-Feb-11 23:53 
QuestionCan I specify which IFilter to use? PinmemberDThrasher1-Mar-10 8:19 
GeneralDoes not work for DOCX files on 64-bit Windows Pinmemberykara14-Jan-10 19:07 
GeneralRe: Does not work for DOCX files on 64-bit Windows PinmemberTom Miller8531-Jan-11 8:14 
GeneralRe: Does not work for DOCX files on 64-bit Windows Pinmemberalex_zero7-Feb-11 0:08 
GeneralRe: Does not work for DOCX files on 64-bit Windows PinmemberMartin_Dann19-May-11 3:31 
QuestionWhy not use the LoadIFilter function to get IFilter? Pinmemberrohit_lodha15-Jul-09 22:23 
Generalvery cool stuff but i have some problems with html\xml files Pinmemberickeoderwat1-Jul-09 9:56 
GeneralDoes not load filter for .docx file under Vista Pinmemberhendi610-Apr-09 12:34 
AnswerRe: Does not load filter for .docx file under Vista Pinmemberhendi67-May-09 7:36 
GeneralComparision to Eyal Post article [modified] PinmemberPetr Felzmann4-Mar-09 22:46 
GeneralRe: Comparision to Eyal Post article Pinmemberalex_zero5-Mar-09 16:18 
QuestionIs there version for visual studio 2005? Pinmemberfsben3-Feb-09 6:25 
AnswerRe: Is there version for visual studio 2005? Pinmemberalex_zero5-Mar-09 16:32 
QuestionWhat about properties of *.doc Pinmembertprakash13-Jan-09 0:22 
AnswerRe: What about properties of *.doc [modified] Pinmemberalex_zero5-Mar-09 16:57 
GeneralGood job Pinmemberwooting12-Jan-09 22:49 
GeneralRe: Good job Pinmemberalex_zero5-Mar-09 16:33 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.1411019.1 | Last Updated 9 Feb 2011
Article Copyright 2009 by alex_zero
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid