YaCy P2P Search Sample in C#

• Download sample - 33.7 KB

Introduction

This is a quick example of how to query a YaCy peer for search results using C#.

Background

YaCy is an Open Source peer to peer web search engine/crawler. It can be used to provide search functionality for local intranets and document stores, or it can contribute to a global P2P network to provide web crawling/indexing/search in a decentralized fashion.

Read more at http://yacy.net.

Using the Code

The YacyServer class (below) is a simple class that keeps track of the URI of the YaCy peer you wish to use and details of your web proxy (if needed).

The URI passed to the constructor should be an active YaCy peer.

The main method used is Search, which has an override to just use the default maximum number of results. As you can see, it's just a web request, and the results are RSS.

The return result is an enumeration of SearchResult objects.

/// <summary>
/// Encapsulates a YaCy web server
/// </summary>
public class YacyServer
{
    private Uri uri;
    private IWebProxy proxy;

    public YacyServer(Uri webServerUri) : this(null, webServerUri) { }

    public YacyServer(IWebProxy proxy, Uri webServerUri)
    {
        uri = webServerUri;
        this.proxy = proxy;            
    }

    private int _max = 500;
    public int MaxResultDefault
    {
        get { return _max; }
        set { if (value != 0) _max = Math.Abs(value); }
    }

    public IEnumerable<SearchResult> Search(string search)
    {
        return Search(0, search);
    }

    public IEnumerable<SearchResult> Search(int maxResults, string search)
    {
        List<SearchResult> result = new List<SearchResult>();

        string request = uri.Scheme + "://" + uri.Host + ":" + uri.Port + 
                         "/yacysearch.rss?query=" + HttpUtility.UrlEncode(search);
        if(maxResults > 0)
            request += "&maximumRecords=" + maxResults.ToString();
        else
            request += "&maximumRecords=" + _max.ToString();

        request += "&resource=global";
        // important, don't just search the local index

        WebRequest wr = (WebRequest)HttpWebRequest.Create(request);
        if (proxy != null)
            wr.Proxy = proxy;

        WebResponse resp = wr.GetResponse();
        XmlDocument doc = new XmlDocument();
        doc.Load(resp.GetResponseStream());

        XmlNodeList results = doc.SelectNodes("/rss/channel/item");

        foreach (XmlNode node in results)
            result.Add(new SearchResult(node));

        return (IEnumerable<SearchResult>)result;
    }
}

Obviously, there are a number of things not shown: error handling, or making the web request asynchronous, for example.

The SearchResult class is just a wrapper:

/// <summary>
/// Encapsulates a YaCy search result
/// </summary>
public class SearchResult
{
    private XmlNode data;

    protected internal SearchResult(XmlNode data)
    {
        this.data = data;
    }

    public string Title
    {
        get { return data["title"].InnerText; }
    }

    public string Link
    {
        get { return data["link"].InnerText; }
    }

    public string Description
    {
        get { return data["description"].InnerText; }
    }
}

Using these classes in a Windows Form goes something like this:

private void Form1_Load(object sender, EventArgs e)
{
    // this is my yacy peer address,
    // read how to set up your own yacy peer at http://yacy.net
    server = new YacyServer(new Uri("http://abbot.is-a-geek.net:8080"));
}

private YacyServer server;
private void button1_Click(object sender, EventArgs e)
{
    button1.Enabled = false;
    IEnumerable<SearchResult> result = server.Search(textBox1.Text);

    // The search result uses this interface
    // so it can be further filtered using Linq
    // eg:
    // var x = from p in result where p.Title.Contains("Blog") select p;

    dataGridView1.DataSource = result;

    button1.Enabled = true;
}

Points of Interest

The Search function returns IEnumerable<T>, so the search results could be filtered further on the client using LINQ.

For further API documentation, click here.

Summary

YaCy has a large API and can search many types of content; if anyone wants to request further sample YaCy articles, I will do my best to oblige.

YaCy offers an Open Source, peer to peer search solution which can be used in many ways to improve your own website as well as the freedom of the internet (YaCy can't be censored as there is no central point of control).

The public peer network ('freeworld') is still growing, please get involved.