Click here to Skip to main content
Click here to Skip to main content

Batch Converting Linux ASCII Files to Windows

, 7 Dec 2004
Rate this:
Please Sign up or sign in to vote.
This article explains using the Drag and Drop events to filter for files from the system. It also converts line ends from Linux (or other operating systems) to Windows style \n (new line) \r (carriage return) line ends.

Sample Image

Introduction

This article is a simple exercise in Drag & Drop, TextFile IO, and Batching.

Background

So my business partner and I were migrating our DNS information from BIND on one of our Linux boxes to a new Windows 2003 machine. In the documentation for Windows DNS, we found that you could simply move the Zone files created by BIND for our DNS Zones to the Windows DNS directory, rename them, and voila, you have a copy of DNS records.

As with all things that sound simple, they usually aren't. Windows didn't like the formatting of the Linux BIND files because they had only the \n for line feed / carriage return, and Windows wanted more. Windows uses \n\r for new lines. Well, I knew there were probably utilities out there for converting Linux (and other) ASCII files to Windows style ASCII files; however, I figured, what the heck, I'll write my own.

So I set upon writing this little utility. It took about 20 minutes, and I decided to make this my first post for Code Project.

The Code

The code is quite simple, but it does explain about doing drag & drop operations and filtering for files. Also included is some simple RegEx matching for characters in the UTF-8 character set. I have commented on the code profusely, so there shouldn't be too many questions.

The first thing we needed was a drag and drop interface, because we didn't want to open the files manually or by using a file dialog. So we created a list view with three columns: File (name), Size (bytes), and Status.

We then set up the list view with event handlers on DragEnter and DragDrop.

private void DropSpot_DragEnter(object sender, System.Windows.Forms.DragEventArgs e)
{
    // We only want to accept files, so we only set our DragDropEffects 
    // if that's what's being dragged in
    if (e.Data.GetDataPresent(DataFormats.FileDrop, false)==true)
    {
        e.Effect = DragDropEffects.All;
    }
}
ArrayList Files = new ArrayList();
private void DropSpot_DragDrop(object sender, 
                System.Windows.Forms.DragEventArgs e)
{
    // Get a list of all objects in the Drop Data, that are files
    string[] files = (string[])e.Data.GetData(DataFormats.FileDrop);
    // Iterate through the dropped files
    for (int i=0;i<files.Length;i++)
    {
        // Add the to our ArrayList
        Files.Add(files[i]);
        // Create our new List View item
        ListViewItem item = new ListViewItem();
        // Get a file info object
        // we use this for getting file size, etc.
        System.IO.FileInfo fInfo = new System.IO.FileInfo(files[i]);
        item.Text = System.IO.Path.GetFileName(fInfo.Name);
        item.SubItems.Add(fInfo.Length.ToString());
        item.SubItems.Add("Pending");
        FileListView.Items.Add(item);
        FileListView.Tag = Files[Files.Count-1];
    }
    // Refresh the file list - for good measure
    this.Refresh();
    // If we added files, clear the instruction label
    if (FileListView.Items.Count>0) label1.Visible = false;

}

In both handlers, you can see that we filter using the DataFormats enum. We only want to accept objects of type DataFormats.FileDrop. This gives us the objects that are files being dragged from a folder, or equivalent system objects.

The first handler, DragEnter, simply sets the DragDropEffects which will change the the mouse cursor appropriately, indicating that files can be dropped here.

Incidentally, for purposes of user direction, I added a label on top of the list view, which instructs users to "Drag and Drop files here to repair". This label is also given the same two event handlers described above. The label will "disappear" when files are added, and reappear when the file list is cleared.

The next piece of code needed was something to determine (as accurately as possible) whether or not the file is plain ASCII text or binary.

Note: By definition, all files stored on a computer are stored in binary format. What we are seeking to determine is whether or not the file stored is one containing "text" or "ASCII", and not binary data such as an image or otherwise non-textual files. After much research on other projects, I have yet to find a method that is 100% accurate, but the method described below is quite accurate. It requires that the characters in the file all fall within the UTF-8 character set.

On to the code. This method, VerifyAscii(string Buffer) will take the input buffer, and, using C#'s Regular Expression matching, will search the file in blocks, to find whether or not all of the characters within these blocks are ASCII compliant. Note that the RegEx is set to \xFF which can be changed to \x80 for the 7-bit ASCII set.

private bool VerifyAscii(string Buffer)
{
    // Create Regex for matching only the Ascii Table
    System.Text.RegularExpressions.Regex R = 
            new System.Text.RegularExpressions.Regex("[\x00-\xFF]");
    // The Size of the block that we want to analyze
    // Done this way for performance
    // Much overhead (depending on size of file) to Regex the whole thing
    int BlockSize = 10;
    // Our Iteration variables
    int Start;
    int Len;
    string Block;
    System.Text.RegularExpressions.MatchCollection matchColl;
    // Iterate through our buffer
    for (int i=0;i<(Buffer.Length/BlockSize);i++)
    {
        // Starting Point for this iteration
        Start = (i*5);
        // Ternerary operator used to assign length of this block
        // we don't want to overshoot the end of the string buffer
        Len = (Start+BlockSize>Buffer.Length) ? (Buffer.Length-Start) : BlockSize;
        // Get our block from the buffer
        Block  = Buffer.Substring(Start,Len);
        // Run our Regex, and get our match collection
        matchColl = R.Matches(Block);
        // If our match count is less that the length of the string,
        // we know that we have characters outside of the ascii table
        if (matchColl.Count<Len)
        {
            // Return false, this buffer could not be
            // evaluated as Ascii Only
            return false;
        }
    }
    // No bad charaters were found, 
    // so all characters are within the ascii table
    // Return true
    return true;
}

What's Going On: For performance, the buffer is split into chunks. Doing a RegEx on a string that was the size of a typical system log file, for example, would take a long time, and it would be a lot of overhead. Doing this portion in small chunks allows for faster processing of the regular expression matching, as well as the re-creation of the string through each iteration.

Note: I opted not to use StringBuilder here, as this just started out as a quick utility for converting about 200 DNS zone files; but that would be the way to go in place of the string buffer. In C#, when working with strings, you are dealing with a special type. C# treats strings as value types, but behind the scenes, they are objects. When you perform virtually any operation on a string such as concatenation, or truncation, the original object is not modified, a new object is created, the data is moved, and the old object is destroyed. This can create a lot of overhead in an intensive application.

Lesson: When performing large amounts of string operations in a real application, use System.Text.StringBuilder as it is designed for these purposes.

Next up, we have our file repair method aptly named, RepairFile(string Path).

private bool RepairFile(string Path)
{
    // Create a file info object
    System.IO.FileInfo Fi = new System.IO.FileInfo(Path);
    // If the file exists, proceed
    if (Fi.Exists)
    {
        // NOTE: Error trapping omitted for 
        // readability
        // You would want to trap the file operations in
        // a try / catch / finally block 
        // -----------------------------------------------
        // Create a StreamReader object 
        // We use a StreamReader because we are assuming 
        // that we are dealing with a text file
        System.IO.StreamReader S = Fi.OpenText();
        // Read the entire file -
        // NOTE: This would be better done using buffering
        // for performance, but for this example, I omitted it
        string FileBuffer = S.ReadToEnd();
        // Close our reader
        S.Close();
        // Call to our VerifyAscii method to ensure that
        // this is NOT a binary file
        if (VerifyAscii(FileBuffer))
        {
            // Split our buffer into lines
            string[] Lines = FileBuffer.Split('\n');
            // Create our StreamWriter
            // Again, using a streamWriter, since we are
            // dealing with Text
            System.IO.StreamWriter W = 
                   new System.IO.StreamWriter(Fi.OpenWrite());
            // Loop through our "Lines" and use the StreamWriter's WriteLine
            // Method to terminate the lines with the operating system
            // specific carriage return / line feed combination
            for (int i=0;i<Lines.Length;i++)
            {
                W.WriteLine(Lines[i].Trim());
            }
            // Close our writer
            W.Close();
            return true;
        }
        else
        {
            // Error Message for "non-ascii" files
            MessageBox.Show(Path+" \nDoes not Appear to be plain text. " + 
                       " No repair will be performed","File Format Error");
            return false;
        }
    }
    return false;
}

What's Going on: This code is fairly self explanatory and, in places where it is not, the code contains a plethora of comments. In summary, this method opens the file specified by "path" using a StreamReader object. We use a StreamReader object because we are assuming text in the files we're reading, and it is better suited for this purpose than the plain Stream object.

Note: For simplicity, the file is read in its entirety using the ReadToEnd method of the StreamReader object. In a real world application, it would be better for performance of the application to read the file in, using blocks.

Once the file is loaded, it is first sent to our VerifyAscii method described above. If VerifyAscii returns true, the buffer is split into lines using String.Split method on the new-line character (\n). A StreamWriter object is then created by opening our original file in Write mode. For preservation purposes, it may be better to use a different file for the output, but I wasn't worried about the possible damage as these were copies anyway.

We then iterate through row in the Lines[] Array and write it back to the original file. By using StreamWriter.WriteLine(), we are terminating the lines with the operating system specific line termination characters. In Windows, lines are terminated by \n\r (ASCII 13 + ASCII 10).

Bringing it all together

The files, once dragged, are added to the ListView in the DragDrop event handler. To start the "repair", the "Go" button is clicked, and processing begins.

private void btnGo_Click(object sender, System.EventArgs e)
{
    // Iterate through our files list
    for (int i=0;i<Files.Count;i++)
    {                
        // If repair was successful, 
        // Mark the status column for this file complete,
        // otherwise mark it failed
        if (!RepairFile(Files[i].ToString()))
        {
            if (i<FileListView.Items.Count)
            {
                FileListView.Items[i].SubItems[2].Text = "Failed";
            }
        }
        else
        {
            if (i<FileListView.Items.Count)
            {
                FileListView.Items[i].SubItems[2].Text = "Complete";
            }
        }
    }
}

What's Going on: Earlier, in our DragDrop event handler, files were added to our list view, and an ArrayList. When our "Go" button is clicked, we iterate through the objects (file paths, in this case) stored in the ArrayList. Each item is successively passed to the RepairFile method and, if RepairFile returns true, the corresponding item in the list view is marked "Complete"; otherwise, an error message is displayed and the item is marked "Failed".

Summary

Demonstrated here was the use of Drag and Drop event handlers to filter Drag Objects as files from the system, the use of Regex to match the UTF-8 character set, and batch processing of files. There are certainly areas in which this code could be improved, but many of these areas I left up to you to discover, and to keep the code simple.

Using the code

Here is the complete code. This was coded in Visual Studio 2003, so you should simply open and build it similarly.

using System;
using System.Drawing;
using System.Collections;
using System.ComponentModel;
using System.Windows.Forms;

namespace LinWinRepair
{
    /// <summary>
    /// Summary description for Form1.
    /// </summary>
    public class mainUI : System.Windows.Forms.Form
    {
        private System.Windows.Forms.Panel panel1;
        private System.Windows.Forms.Button btnClear;
        private System.Windows.Forms.Button btnGo;
        private System.Windows.Forms.ListView FileListView;
        private System.Windows.Forms.Label label1;
        private System.Windows.Forms.ColumnHeader columnHeader4;
        private System.Windows.Forms.ColumnHeader columnHeader5;
        private System.Windows.Forms.ColumnHeader columnHeader6;
        /// <summary>
        /// Required designer variable.
        /// </summary>
        private System.ComponentModel.Container components = null;

        public mainUI()
        {
            //
            // Required for Windows Form Designer support
            //
            InitializeComponent();

            //
            // TODO: Add any constructor code after InitializeComponent call
            //
        }

        /// <summary>
        /// Clean up any resources being used.
        /// </summary>
        protected override void Dispose( bool disposing )
        {
            if( disposing )
            {
                if (components != null) 
                {
                    components.Dispose();
                }
            }
            base.Dispose( disposing );
        }

        #region Windows Form Designer generated code
        /// <summary>
        /// Required method for Designer support - do not modify
        /// the contents of this method with the code editor.
        /// </summary>
        private void InitializeComponent()
        {
            this.panel1 = new System.Windows.Forms.Panel();
            this.btnClear = new System.Windows.Forms.Button();
            this.btnGo = new System.Windows.Forms.Button();
            this.FileListView = new System.Windows.Forms.ListView();
            this.label1 = new System.Windows.Forms.Label();
            this.columnHeader4 = new System.Windows.Forms.ColumnHeader();
            this.columnHeader5 = new System.Windows.Forms.ColumnHeader();
            this.columnHeader6 = new System.Windows.Forms.ColumnHeader();
            this.panel1.SuspendLayout();
            this.SuspendLayout();
            // 
            // panel1
            // 
            this.panel1.Controls.Add(this.btnClear);
            this.panel1.Controls.Add(this.btnGo);
            this.panel1.Dock = System.Windows.Forms.DockStyle.Bottom;
            this.panel1.Location = new System.Drawing.Point(15, 303);
            this.panel1.Name = "panel1";
            this.panel1.Size = new System.Drawing.Size(370, 40);
            this.panel1.TabIndex = 1;
            // 
            // btnClear
            // 
            this.btnClear.Anchor = ((System.Windows.Forms.AnchorStyles)
                 ((System.Windows.Forms.AnchorStyles.Bottom | 
                 System.Windows.Forms.AnchorStyles.Right)));
            this.btnClear.Location = new System.Drawing.Point(208, 5);
            this.btnClear.Name = "btnClear";
            this.btnClear.TabIndex = 6;
            this.btnClear.Text = "Clear";
            this.btnClear.Click += new System.EventHandler(this.btnClear_Click);
            // 
            // btnGo
            // 
            this.btnGo.Anchor = ((System.Windows.Forms.AnchorStyles)
                 ((System.Windows.Forms.AnchorStyles.Bottom | 
                 System.Windows.Forms.AnchorStyles.Right)));
            this.btnGo.Location = new System.Drawing.Point(288, 5);
            this.btnGo.Name = "btnGo";
            this.btnGo.TabIndex = 5;
            this.btnGo.Text = "Go";
            this.btnGo.Click += new System.EventHandler(this.btnGo_Click);
            // 
            // FileListView
            // 
            this.FileListView.AllowDrop = true;
            this.FileListView.Columns.AddRange(new 
                 System.Windows.Forms.ColumnHeader[] {
                        this.columnHeader4,
                        this.columnHeader5,
                        this.columnHeader6});
            this.FileListView.Dock = System.Windows.Forms.DockStyle.Fill;
            this.FileListView.Location = new System.Drawing.Point(15, 15);
            this.FileListView.Name = "FileListView";
            this.FileListView.Size = new System.Drawing.Size(370, 288);
            this.FileListView.TabIndex = 0;
            this.FileListView.View = System.Windows.Forms.View.Details;
            this.FileListView.DragDrop += new 
                 System.Windows.Forms.DragEventHandler(this.DropSpot_DragDrop);
            this.FileListView.DragEnter += new 
                 System.Windows.Forms.DragEventHandler(this.DropSpot_DragEnter);
            // 
            // label1
            // 
            this.label1.AllowDrop = true;
            this.label1.BackColor = System.Drawing.SystemColors.Window;
            this.label1.Location = new System.Drawing.Point(24, 136);
            this.label1.Name = "label1";
            this.label1.Size = new System.Drawing.Size(360, 23);
            this.label1.TabIndex = 2;
            this.label1.Text = "Drag and Drop Files here to Repair";
            this.label1.TextAlign = System.Drawing.ContentAlignment.MiddleCenter;
            this.label1.DragEnter += new 
                 System.Windows.Forms.DragEventHandler(this.DropSpot_DragEnter);
            this.label1.DragDrop += new 
                 System.Windows.Forms.DragEventHandler(this.DropSpot_DragDrop);
            // 
            // columnHeader4
            // 
            this.columnHeader4.Text = "File";
            this.columnHeader4.Width = 218;
            // 
            // columnHeader5
            // 
            this.columnHeader5.Text = "Size (bytes)";
            this.columnHeader5.Width = 87;
            // 
            // columnHeader6
            // 
            this.columnHeader6.Text = "Status";
            // 
            // mainUI
            // 
            this.AutoScaleBaseSize = new System.Drawing.Size(5, 13);
            this.ClientSize = new System.Drawing.Size(400, 358);
            this.Controls.Add(this.label1);
            this.Controls.Add(this.FileListView);
            this.Controls.Add(this.panel1);
            this.DockPadding.All = 15;
            this.Name = "mainUI";
            this.Text = "LinWin File Repair Tool";
            this.panel1.ResumeLayout(false);
            this.ResumeLayout(false);

        }
        #endregion

        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]
        static void Main() 
        {
            Application.Run(new mainUI());
        }

        private void DropSpot_DragEnter(object sender, 
                System.Windows.Forms.DragEventArgs e)
        {
            // We only want to accept files, so we only set our DragDropEffects 
            // if that's what's being dragged in
            if (e.Data.GetDataPresent(DataFormats.FileDrop, false)==true)
            {
                e.Effect = DragDropEffects.All;
            }
        }
        ArrayList Files = new ArrayList();
        private void DropSpot_DragDrop(object sender, 
                System.Windows.Forms.DragEventArgs e)
        {
            // Get a list of all objects in the Drop Data, that are files
            string[] files = (string[])e.Data.GetData(DataFormats.FileDrop);
            // Iterate through the dropped files
            for (int i=0;i<files.Length;i++)
            {
                // Add the to our ArrayList
                Files.Add(files[i]);
                // Create our new List View item
                ListViewItem item = new ListViewItem();
                // Get a file info object
                // we use this for getting file size, etc.
                System.IO.FileInfo fInfo = new System.IO.FileInfo(files[i]);
                item.Text = System.IO.Path.GetFileName(fInfo.Name);
                item.SubItems.Add(fInfo.Length.ToString());
                item.SubItems.Add("Pending");
                FileListView.Items.Add(item);
                FileListView.Tag = Files[Files.Count-1];
            }
            // Refresh the file list - for good measure
            this.Refresh();
            // If we added files, clear the instruction label
            if (FileListView.Items.Count>0) label1.Visible = false;

        }

        private void btnClear_Click(object sender, System.EventArgs e)
        {

            // Clear our ArrayList
            Files.Clear();
            // Clear our File ListView
            FileListView.Clear();
            // Bring the old instruction label back
            label1.Visible=true;
        }

        private void btnGo_Click(object sender, System.EventArgs e)
        {    
            // Iterate through our files list
            for (int i=0;i<Files.Count;i++)
            {
                // If repair was successful, 
                // Mark the status column for this file complete,
                // otherwise mark it failed
                if (!RepairFile(Files[i].ToString()))
                {
                    if (i<FileListView.Items.Count)
                    {
                        FileListView.Items[i].SubItems[2].Text = "Failed";
                    }
                }
                else
                {
                    if (i<FileListView.Items.Count)
                    {
                        FileListView.Items[i].SubItems[2].Text = "Complete";
                    }
                }
            }
        }

        private bool RepairFile(string Path)
        {
            // Create a file info object
            System.IO.FileInfo Fi = new System.IO.FileInfo(Path);
            // If the file exists, proceed
            if (Fi.Exists)
            {
                // NOTE: Error trapping omitted for 
                // readability
                // You would want to trap the file operations in
                // a try / catch / finally block 
                // -----------------------------------------------
                // Create a StreamReader object 
                // We use a StreamReader because we are assuming 
                // that we are dealing with a text file
                System.IO.StreamReader S = Fi.OpenText();
                // Read the entire file -
                // NOTE: This would be better done using buffering
                // for performance, but for this example, I omitted it
                string FileBuffer = S.ReadToEnd();
                // Close our reader
                S.Close();
                // Call to our VerifyAscii method to ensure that
                // this is NOT a binary file
                if (VerifyAscii(FileBuffer))
                {
                    // Split our buffer into lines
                    string[] Lines = FileBuffer.Split('\n');
                    // Create our StreamWriter
                    // Again, using a streamWriter, since we are
                    // dealing with Text
                    System.IO.StreamWriter W = 
                        new System.IO.StreamWriter(Fi.OpenWrite());
                    // Loop through our "Lines"
                    // and use the StreamWriter's WriteLine
                    // Method to terminate
                    // the lines with the operating system
                    // specific carriage return / line feed combination
                    for (int i=0;i<Lines.Length;i++)
                    {
                        W.WriteLine(Lines[i].Trim());
                    }
                    // Close our writer
                    W.Close();
                    return true;
                }
                else
                {
                    // Error Message for "non-ascii" files
                    MessageBox.Show(Path+" \nDoes not Appear to be plain text. " + 
                               " No repair will be performed","File Format Error");
                    return false;
                }
            }
            return false;
        }

        private bool VerifyAscii(string Buffer)
        {
            // Create Regex for matching only the Ascii Table
            System.Text.RegularExpressions.Regex R = 
                new System.Text.RegularExpressions.Regex("[\x00-\xFF]");
            // The Size of the block that we want to analyze
            // Done this way for performance
            // Much overhead (depending on size of file) to Regex the whole thing
            int BlockSize = 10;
            // Our Iteration variables
            int Start;
            int Len;
            string Block;
            System.Text.RegularExpressions.MatchCollection matchColl;
            // Iterate through our buffer
            for (int i=0;i<(Buffer.Length/BlockSize);i++)
            {
                // Starting Point for this iteration
                Start = (i*5);
                // Ternerary operator used to assign length of this block
                // we don't want to overshoot the end of the string buffer
                Len = 
                  (Start+BlockSize>Buffer.Length) ? (Buffer.Length-Start) : BlockSize;
                // Get our block from the buffer
                Block  = Buffer.Substring(Start,Len);
                // Run our Regex, and get our match collection
                matchColl = R.Matches(Block);
                // If our match count is less that the length of the string,
                // we know that we have characters outside of the ascii table
                if (matchColl.Count<Len)
                {
                    // Return false, this buffer could not be
                    // evaluated as Ascii Only
                    return false;
                }
            }
            // No bad charaters were found, 
            // so all characters are within the ascii table
            // Return true
            return true;
        }
    
    }
}

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Seth Webster

United States United States
Currently, I am the President of a medium size Tucson, Az based development / consulting firm. I have been developing solutions in various languages for over 15 years. Like many, I am a tech junkie.
 
Daily, I oversee and participate in development on projects ranging from simple online presence deployment, to full scale systems integration and management infrastructure.
 
Languages of choice are PHP & MySql solutions, C# / ASP.NET with SQL Server.
 
Prior to opening the doors for my company, Webglimmer Internet Solutions, I made a living (with development / consulting on the side) as a musician. For 11 years, I played everything from Rock to Blues, touring the U.S., Canada and
Mexico. Unfortunately, Musicians make the same money they did 100 years ago. Unless of course, you "make it". Alas, music never paid the bills.

Comments and Discussions

 
Generalunix2dos Pinmemberrequesttimedout17-Jun-08 15:51 
GeneralRe: unix2dos PinmemberTrollslayer24-Jun-08 4:20 
GeneralQuestion Pinmembersheitman808-Dec-04 12:30 
GeneralRe: Question PinmemberSeth Webster8-Dec-04 12:45 
Thanks for the input! Smile | :) I am not sure whether or not using the String.Replace method would be faster or not. It would probably be faster if using a StringBuilder.Replace as you only deal with one object. If using the String.Replace method, a new string object would be created each time a replace was done on one of the \n characters, and that new String would be re-scanned for \n instances. So I think with a basic string, that the method I used is the faster way.
 
The best method may be to use a StringBuilder, but as I stated in the article, I used a basic string for simplicity. StringBuilder isn't much more complicated to use, though I think it would take longer on the RegEx matching, so a real test would be needed to determine which is really faster.
 
Thanks a lot for the response.
 


 
Seth Webster

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140721.1 | Last Updated 8 Dec 2004
Article Copyright 2004 by Seth Webster
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid