Click here to Skip to main content
Click here to Skip to main content

OmniSearch (Google Caching Engine)

, 19 Feb 2003
Rate this:
Please Sign up or sign in to vote.
OmniSearch attempts at showing a way to reduce the number of hits to the Google Web Service by Caching Searches.
<!-- Article Starts - DO NOT ADD HTML/BODY START TAGS--> <!-- Download Links --> <!-- Add the rest of your HTML here -->

Introduction

The OmniSearch project was founded around the idea of designing a customizable search engine for seamless integration into any current website that run the ASP.NET Framework.  OmniSearch can accomplish this by caching the results from the Google API and displaying the results as a manipulated XML file using XSL.  This will allow for flexibility and ease of programming for the Administrator.

OmniSearch will use the Google API as the backbone to the OmniSearch Engine, which will do the bulk of the processing for the application.  OmniSearch’s fully query-able statistics engine will collect click-through habits, from the users, for later analysis by the Administrator of the site.  The OmniSearch engine will serve a variety of functions including, but not limited to:

  • Maintaining a statistical record of click-throughs.
  • Caching search results from Google’s XML SOAP API.
  • Providing a Wireless WAP interface for mobile users to take advantage of OmniSearch.
  • Provide components to integrate into existing ASP.Net projects that webmasters may be running.

Background

This project’s main goal is to get around the limitations that the Google API forces on its users, which only allows 1000 hits per day.  Our team plans to get around this limitation by caching the serialized results from the Google API into an XML file.  Subsequent web searches will first query the XML Cache to find previous search results that will fulfill the current query of the current user.  By utilizing the cache, OmniSearch will become quicker and better able to handle user search requests, the more that it is being used by the sites users. 

One of the hardest things about being a website administrator is collecting useful information about users habits on your site.  So to further challenge the OmniSearch team we decided to add a click-through logging feature that will enable administrators to view pages that may be getting high traffic load.  This will also aid in decreasing the time needed to find certain pages, because the administrator will be able to tell the users such things as “The Site of the Day”, “Top 5 Sites for this Month”, and so on.

In Addition to the cached search results and statistics database, OmniSearch will also support WAP interfaces for the search engine.  By using ASP.Net Mobile Architecture, the OmniSearch team is able to provide a wireless interface that will customize it self according to the wireless browser that is being used.

Using the code

The following parameters need to be setup in the web.config file, before anything will work correctly.

<!--<span class="code-comment"> application specific settings --></span>
<appSettings>
    <!--<span class="code-comment"> Realative Location of Cache --></span>
    <add key="CacheLocation" value="\cache" />
    <!--<span class="code-comment"> Directory Name of Website --></span>
    <add key="DirectoryName" value="\OmniSearch" />
    <!--<span class="code-comment"> Static Location of Website --></span>
    <add key="WebSiteLocation" value="D:\Websites\Development\OmniSearch" />
    <!--<span class="code-comment"> Proxy Settings if needed --></span>
    <!--<span class="code-comment"> Set a host to use as an HTTP proxy. If unset, GoogleSearch will also check the 
        system properties and use those values. If those are also unset, HTTP requests 
        go direct with no proxy.
    --></span>
    <add key="ProxyHost" value="" />
    <!--<span class="code-comment"> Set a port to use as an HTTP proxy. Only used if proxyHost is also set. If 
        unset, port 80 is assumed to be the default.
    --></span>
    <add key="ProxyPort" value="" />
    <!--<span class="code-comment"> Set the username required for the HTTP proxy. Only used if ProxyHost is also set. --></span>
    <add key="ProxyUsername" value="" />
    <!--<span class="code-comment"> Set the password required for the HTTP proxy. Only used if ProxyHost is also set. --></span>
    <add key="ProxyPassword" value="" />
    <!--<span class="code-comment"> Google Configuration Information --></span>
    <!--<span class="code-comment"> Set the user key used for authorization by Google SOAP server. This is a 
        mandatory attribute for all requests.
        A key can be obtained from http://www.google.com/apis/index.html
    --></span>
    <add key="GoogleKey" value="" />
    <!--<span class="code-comment"> Set the maximum number of results to be returned.  
        The number must be between 1 and 10 
    --></span>
    <add key="GoogleMaxReturn" value="10" />
    <!--<span class="code-comment"> Enable or disable the "related-queries" filter.
        Must be [True | False]
    --></span>
    <add key="GoogleFilter" value="False" />
    <!--<span class="code-comment"> Set the restrict. 
        This allows you to restrict the search to a specific document store such as 
        "Penn State", "IST" or any argument that can be used in a normal Google Search.
    --></span>
    <add key="GoogleRestrict" value="" />
    <!--<span class="code-comment"> Enable or disable SafeSearch. When SafeSearch is turned on, sites and web pages 
        containing pornography and explicit sexual content are blocked from search 
        results. While no filter is 100% accurate, Google's filter uses advanced 
        proprietary technology that checks keywords and phrases, URLs and Open Directory 
        categories.
        Must be [True | False]
    --></span>
    <add key="GoogleSafeSearch" value="False" />
    <!--<span class="code-comment"> Domain to search --></span>
    <add key="DomainToSearch" value="ist.psu.edu" />
    <!--<span class="code-comment"> Number of days the cache will stay in memory --></span>
    <add key="ExpireTime" value="5" />
</appSettings>

The following is an example of the XSL file that can be used against the serialized form of the XML Cached Search from Google's Web Service. This XSL file displays the search contents just like you would see on the Google Search Page.  In the source code this file is called searchresults.xsl.

<?xml version="1.0" encoding="utf-8" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" version="4.0" encoding="iso-8859-1" indent="yes" />
<xsl:template match="/">
    <xsl:apply-templates select="GoogleSearchResult" />
</xsl:template>
<xsl:template match="GoogleSearchResult">
    <table width="100%" border="0" cellspacing="0" cellpadding="0" bgcolor="#6699cc" 
        style="padding: 5px, 5px, 5px, 5px; margin-top: 5px;" ID="Table1">
        <tr style="color:white;">
            <td aling="left">Search <b>ist.psu.edu</b> for <b>
                    <xsl:value-of select="searchQuery" />
                </b></td>
            <td align="right">Results <b>
                    <xsl:value-of select="startIndex" />
                </b> - <b>
                    <xsl:value-of select="endIndex" />
                </b> of about <b>
                    <xsl:value-of select="estimatedTotalResultsCount" />
                </b>.</td>
        </tr>
    </table>
    <table border="0" cellspacing="0" cellpadding="0" ID="Table2">
        <xsl:for-each select="resultElements/ResultElement">
            <tr>
                <td>
                    <b>
                        <a style="color:blue;" href="/OmniSearch/forward.aspx?url={URL}">
                            <xsl:value-of select="title" />
                        </a>
                    </b>
                </td>
            </tr>
            <tr>
                <td>
                    <xsl:value-of select="snippet" />
                </td>
            </tr>
            <tr>
                <td>
                    <span style="color:green;"><xsl:value-of select="URL" />
                        - <xsl:value-of select="cachedSize" /></span>
                    <a style="color:gray;" 
                        href="http://216.239.53.100/search?q=cache%3A{URL}+{searchQuery}">
                    Cached</a>
                    <a style="color:gray;" 
                        href="http://www.google.com/search?q=related%3A{URL}">
                    Similar pages</a>
                </td>
            </tr>
            <tr>
                <td>
                    <br />
                </td>
            </tr>
        </xsl:for-each>
    </table>
    <table width="100%" border="0" cellspacing="0" cellpadding="0" bgcolor="#6699cc" 
        style="padding: 5px, 5px, 5px, 5px; margin-top: 5px;" ID="Table3">
        <tr style="color:white;">
            <td align="left">
                <b>
                    <a style="color:white;" 
                        href="/OmniSearch/search.aspx?search={searchQuery}&back={startIndex}">
                        « Back</a>
                </b>
            </td>
            <td align="right">
                <b>
                    <a style="color:white;" 
                        href="/OmniSearch/search.aspx?search={searchQuery}&forward={endIndex}">
                        Next »</a>
                </b>
            </td>
        </tr>
    </table>
</xsl:template>
</xsl:stylesheet>

The following code is all that is needed inorder to impliment the OmniSearch Engine in any ASP.Net webpage.  It took alot of time but I wanted to make it simple to use, but really powerful at the same time.

protected System.Web.UI.WebControls.Xml resultsXml;            
        
private void Page_Load(object sender, System.EventArgs e)
{
    search = new SiteSearch(Request.QueryString["search"]);

    // if there is no start query then the program assumes that
    // it wants to start the search at the begining
    if (Request.QueryString["forward"] == null && Request.QueryString["back"] == null)
        start = 1;
    else if (Request.QueryString["forward"] != null && Request.QueryString["back"] ==  null) 
        start = Convert.ToInt32(Request.QueryString["forward"]) +1; 
    else if (Request.QueryString["forward"] == null && Request.QueryString["back"] != null)
        start =  Convert.ToInt32(Request.QueryString["back"]) - 
            Convert.ToInt32(ConfigurationSettings.AppSettings["GoogleMaxReturn"]);
    else
        start = 1;
        
    if(start <  1)
        start = 1;

    resultsXml.DocumentSource = search.DoSearch(start);
}

This is an example of what the ASP.Net page can look like when you put it all together. The following line of code is all you really need to impliment the OmniSearch Engine in any ASP.Net application that you would choose.

<asp:xml id="resultsXml" runat="server" TransformSource="searchresults.xsl"></asp:xml>

Points of Interest

I learned a couple major things while implimenting this code, they are listed as follows:

  • Googles Web Service is very powerful but has many limitations beyond the 1000 hits per day.
  • The use of XML Serialization was very useful to the completation of this project.
  • XSL and how it can be used in .Net even if the client browser doesn't support XSL.
  • In order to get around some obsticals in naming the XML Cache files I had to use a MD5 value of the search query plus an indexing number on the end of the MD5 for the specific pages. The file will be in this format [search query (MD5 Value)].[page index].xml when it is written to the /cache directory.

One thing that needs to be done before OmniSearch can be used.

  • The caching directory /cache needs to grant full access to [computer name]\ASPNET. This needs to be done so that OmniSearch can write the XML Cache files to the directory.

I will be eventually integrating this Engine into the OmniPortal Project that I am also developing.  More about OmniPortal can be found at http://www.sourceforge.net/projects/omniportal or http://beta.omniportal.net (Shameless Plug).

History

2/10/2003 - First Release.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Nick Berardi
Web Developer
United States United States
No Biography provided

Comments and Discussions

 
GeneralMining Google Web Services: Building Applications with the Google API PinsussAnonymous21-Aug-05 20:50 
GeneralI don't see how this would help Pinmemberskrichten@comcast.net3-Dec-03 13:51 
GeneralRe: I don't see how this would help PinsussAnonymous3-Dec-03 13:55 
GeneralRe: I don't see how this would help PinmemberSteve Krichten4-Dec-03 3:35 
GeneralRe: I don't see how this would help PinmemberNick Berardi9-Dec-03 6:45 
GeneralMore info PinmemberSean Winstead20-Feb-03 6:06 
GeneralRe: More info Pinmemberleppie20-Feb-03 6:39 
GeneralRe: More info PinmemberNick Berardi20-Feb-03 7:54 
GeneralRe: More info PinmemberSean Winstead20-Feb-03 9:07 
GeneralRe: More info PinmemberJianhan Zhu4-Mar-04 3:15 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140721.1 | Last Updated 20 Feb 2003
Article Copyright 2003 by Nick Berardi
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid