Introduction
Most people are now aware that most of the world is not English, even though it's very easy to miss this fact when surfing the web, simply because Google gives English results to people searching English, and so you conveniently miss all the pages in German/French/Italian etc.
The popularity of Altavista's famous Babelfish service is therefore hardly surprising - converting text or web pages into other languages is a useful thing to do.
For a while, anyone looking to integrate translation into their app would simply have had to plug in the Babelfish WSDL. Posters to newsgroups were directed to the free service from xmethods, a good source for a variety of web services (SMS, etc.). In fact, the Babelfish WSDL is the 9th hit on Google for WSDL.
So I plugged it into my apps, intranet, extranet and anything else that vaguely looked like it would benefit from a translation service. And life was good.
But one day the service stopped working, apparently for good. So I had to write a replacement. And here it is.
Code
This is a pretty simple job, and can be broken down into the following subtasks:
- Get text for translation and encode it into a HTTP POST request
- Send the data to the web server, acting in effect as a .NET web browser
- Read the response back into a big string
- Remove all the HTML and formatting and send the raw translated string back to the client.
So fire up Visual Studio .NET, and create an ASP.NET Web Service, and name it Translation, and add a Translate.asmx file. There are two inputs: the translation mode (e.g., French to English), and the data to be translated (e.g., 'the quick brown fox jumps over the lazy dog'). To make it a plug-in replacement for the old service, I gave my method the same name and parameters as the old one:
[WebMethod]
public string BabelFish(string translationmode, string sourcedata)
{
}
The translation modes can be found in the source of the page at Babelfish:
readonly string[] VALIDTRANSLATIONMODES = new string[]
{"en_zh", "en_fr", "en_de", "en_it", "en_ja", "en_ko", "en_pt", "en_es",
"zh_en", "fr_en", "fr_de", "de_en", "de_fr", "it_en", "ja_en", "ko_en",
"pt_en", "ru_en", "es_en"};
The code performs validation to check for a valid mode before passing it on to Babelfish. After that, we create a POST request. The syntax for a HTTP POST request looks something like this:
POST /babelfish/tr/ HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 51
lp=en_fr&tt=urltext&intl=1&doit=done&urltext=cheese
It's pretty simple, and if you want, you could use low-level sockets to write the data to the server. Microsoft provides some better ways to do this however, and so we use the HttpWebRequest class, which has lots of built-in features to make it easy to work with HTTP connections.
Uri uri = new Uri(BABELFISHURL);
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(uri);
request.Referer = BABELFISHREFERER;
string postsourcedata;
postsourcedata = "lp=" + translationmode +
"&tt=urltext&intl=1&doit=done&urltext=" +
HttpUtility.UrlEncode(sourcedata);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postsourcedata.Length;
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
Stream writeStream = request.GetRequestStream();
UTF8Encoding encoding = new UTF8Encoding();
byte[] bytes = encoding.GetBytes(postsourcedata);
writeStream.Write(bytes, 0, bytes.Length);
writeStream.Close();
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader readStream = new StreamReader (responseStream, Encoding.UTF8);
string page = readStream.ReadToEnd();
We end up with a string containing the entire Babelfish page. As it stands, this is about 99% noise (HTML tags, Altavista information, etc.), and 1% the translation we were looking for. So we need a regular expression to find the translated text. By looking at the HTML page, you will find the translation is contained between:
<Div style=padding:10px; lang=fr>translation here</div>
So the required regular expression looks like this (note: while testing my regular expressions, I got lots of help from Regulator):
<Div style=padding:10px; lang=..>((?:.|\n)*?)</div>
This will match the whole <div>...</div> string. This is a fairly complex regular expression, but basically, the . character matches everything, except for newlines, hence the (.|\n) pattern, which means any character (except newlines) or new lines.
The brackets create a matching group, meaning that the text within the brackets (namely the translation) will be put in its own group at index 1 (index 0 contains the whole match).
The ?: pattern suppresses grouping: () normally creates a matching group: in this case, we are only using the pattern to allow for line breaks in long translations.
Finally *? is a lazy regular expression, matching every character up to the first instance of <div>. (If I had used plain *, the expression would be greedy, and would chomp right up to the LAST </div>.)
Here's the code:
Regex reg = new Regex(@"<Div style=padding:10px; lang=..>(.*?)</div>");
MatchCollection matches = reg.Matches(page);
if (matches.Count != 1 || matches[0].Groups.Count != 2)
{
return ERRORSTRINGSTART + "The HTML returned from Babelfish " +
"appears to have changed. Please check for" +
" an updated regular expression" +
ERRORSTRINGEND;
}
return matches[0].Groups[1].Value;
And subject to error checking, that's it!
Using it
Download the code, and unzip it somewhere. Add a virtual directory called Translation in IIS. Go to /translate.asmx and click Test, and enter some test data (say 'en_fr', and 'cheese'). If it works, you are ready to use it in your web and Windows Forms applications.
To use it in your app, add a Web Reference to the asmx, to the program you want to use it in; Visual Studio will create a proxy reference for you, which you can then use to perform translation.
Here's some sample code-behind:
namespace test
{
using System;
using System.Data;
using System.Drawing;
using System.Web;
using System.Web.UI.WebControls;
using localhost1;
using System.Web.UI.HtmlControls;
public class WebUserControl1 : System.Web.UI.UserControl
{
protected System.Web.UI.WebControls.DropDownList ddTranslationMode;
protected System.Web.UI.WebControls.TextBox txtText;
protected System.Web.UI.WebControls.Label lblTranslation;
protected System.Web.UI.WebControls.Button submitButton;
private void Page_Load(object sender, System.EventArgs e)
{
}
protected void submitButton_Click(object sender, System.EventArgs e)
{
string translationMode =
this.ddTranslationMode.SelectedItem.Value;
string translationText = this.txtText.Text.Trim();
string translation = "";
try
{
Translate tr = new Translate();
translation = tr.BabelFish(translationMode,translationText);
}
catch (Exception exp)
{
translation = "There was an error accessing the server: "
+ exp.Message;
}
this.lblTranslation.Text = translation;
}
}
}
|
|
 |
 | The HTML returned from Babelfish appears to have changed. Please check for an updated regular expression
mario emilio zab | 3:16 14 Jan '09 |
|
|
what i can do with this error
|
|
|
|
|
|
|
What did you do to solve this problem?
|
|
|
|
|
|
|
Did you find a solution for this? if yes pls. share
|
|
|
|
|
|
|
I have solved this just change the URL for bablefish to "http://babelfish.yahoo.com/translate_txt";
|
|
|
|
|
|
|
I have changed the url to http://babelfish.yahoo.com/translate_txt. Still same error is coming. Can you please send updated Transalate.asmx.cs file.
|
|
|
|
|
| WOWOWOW
greenknt | 0:31 23 Nov '08 |
|
|
Hey man, I'm going to use this in my skype bot ^^, I'll be sure to ref you for the translation mod (when I post it here ^^) This will be soo cool. 
|
|
|
|
|
| C# version for BabelFish & Google
Zachary Yates | 9:46 23 Jul '08 |
|
|
First of all, thanks for all the work that went into this, it helped me greatly. I felt like I should contribute something as well. I took the basic code and re-factored it a little bit, here is a C# version for both BabelFish & Google:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Security;
using System.Text;
using System.Text.RegularExpressions;
using System.Web;
namespace Translation
{
public class TranslationMode
{
public TranslationMode() { }
public TranslationMode( TranslationMode mode )
{
SourceLanguage = mode.SourceLanguage;
TargetLanguage = mode.TargetLanguage;
}
public TranslationMode( string sourceLanguage, string targetLanguage )
{
SourceLanguage = sourceLanguage;
TargetLanguage = targetLanguage;
}
public virtual string SourceLanguage { get; set; }
public virtual string TargetLanguage { get; set; }
}
public interface ITranslationService
{
string Translate( TranslationMode mode, string text );
bool CanTranslate( TranslationMode mode );
}
public abstract class BaseScreenScrapingTranslationService : ITranslationService
{
public virtual string Url { get; set; }
public virtual string Referer { get; set; }
public virtual string RegexMatch { get; set; }
public virtual string Name
{
get { return "BaseScreenScraper"; }
}
public virtual TranslationMode CurrentTranslationMode { get; set; }
public virtual string Translate( TranslationMode mode, string text )
{
try
{
// set the current mode
CurrentTranslationMode = mode;
// validate and remove trailing spaces
if (string.IsNullOrEmpty(text)) throw new ArgumentNullException("text");
text = text.Trim();
Uri uri = new Uri(Url);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Referer = Referer;
// Encode the text to be translated
string postSourceData = GetPostSourceData(mode, text);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postSourceData.Length;
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";
using (Stream writeStream = request.GetRequestStream())
{
byte[] bytes = Encoding.UTF8.GetBytes(postSourceData);
writeStream.Write(bytes, 0, bytes.Length);
writeStream.Close();
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader readStream = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
string page = readStream.ReadToEnd();
Regex reg = new Regex(RegexMatch, RegexOptions.IgnoreCase);
MatchCollection matches = reg.Matches(page);
if (matches.Count != 1 || matches[0].Groups.Count != 2)
{
throw new InvalidOperationException(
"The HTML returned from " + Name + " appears to have changed. Please check for an updated regular expression.");
}
return matches[0].Groups[1].Value;
}
// Exceptions
catch (ArgumentNullException ex) { throw GetStandardException(ex); }
catch (ArgumentException ex) { throw GetStandardException(ex); }
catch (WebException ex)
{
throw new Exception("There was a problem connecting to the " + Name + " server.", ex);
}
catch (SecurityException ex)
{
throw new Exception("You do not have permission to make HTTP connections. Please check your assembly's permission settings.", ex);
}
catch (Exception ex) { throw GetStandardException(ex); }
}
public abstract bool CanTranslate( TranslationMode mode );
protected abstract string GetPostSourceData( TranslationMode mode, string text );
protected Exception GetStandardException( Exception innerException )
{
return new Exception("Could not translate using " + Name + ".", innerException);
}
}
public class BabelFishTranslationService : BaseScreenScrapingTranslationService
{
static readonly List<string> s_SupportedModes = new List<string>() {
"en_zh", "en_fr", "en_de", "en_it", "en_ja", "en_ko", "en_pt", "en_es",
"zh_en", "fr_en", "fr_de", "de_en", "de_fr", "it_en", "ja_en", "ko_en",
"pt_en", "ru_en", "es_en" };
public override string Url
{
get { return "http://babelfish.yahoo.com/translate_txt"; }
set { base.Url = value; }
}
public override string Referer
{
get { return "http://babelfish.yahoo.com/"; }
set { base.Referer = value; }
}
public override string RegexMatch
{
get { return @"<div id="\""result\"""><div style="\""padding .6em;\""">((?:.|\n)*?)</div></div>"; }
set { base.RegexMatch = value; }
}
public override string Name
{
get { return "BabelFish"; }
}
public override bool CanTranslate( TranslationMode mode )
{
BabelFishTranslationMode myMode = new BabelFishTranslationMode(mode);
if (!s_SupportedModes.Contains(myMode.ToString()))
{
Trace.TraceWarning("TranslationMode is not supported: " + mode);
return false;
}
return true;
}
protected override string GetPostSourceData( TranslationMode mode, string text )
{
BabelFishTranslationMode myMode = new BabelFishTranslationMode(mode);
return string.Format(
"lp={0}&tt=urltext&intl=1&doit=done&urltext={1}",
myMode.ToString(), HttpUtility.UrlEncode(text));
}
internal class BabelFishTranslationMode : TranslationMode
{
public BabelFishTranslationMode( TranslationMode mode )
{
SourceLanguage = mode.SourceLanguage;
TargetLanguage = mode.TargetLanguage;
}
public override string ToString()
{
return SourceLanguage.Substring(0, 2) + "_" + TargetLanguage.Substring(0, 2);
}
}
}
public class GoogleTranslationService : BaseScreenScrapingTranslationService
{
static readonly List<string> s_SourceLanguages = new List<string>() {
"auto", "ar", "bg", "zh-CN", "hr", "cs", "da", "nl", "en",
"fi", "fr", "de", "el", "hi", "it", "ja", "ko",
"no", "pl", "pt", "ro", "ru", "es", "sv" };
static readonly List<string> s_TargetLanguages = new List<string>() {
"ar", "bg", "zh-CN", "zh-TW", "hr", "cs", "da", "nl", "en",
"fi", "fr", "de", "el", "hi", "it", "ja", "ko",
"no", "pl", "pt", "ro", "ru", "es", "sv" };
public override string Url
{
get
{
if (CurrentTranslationMode != null)
{
return string.Format("http://translate.google.com/translate_t?sl={0}&tl={1}",
CurrentTranslationMode.SourceLanguage, CurrentTranslationMode.TargetLanguage);
}
else
{
return "http://translate.google.com/translate_t";
}
}
set { base.Url = value; }
}
public override string Referer
{
get { return "http://translate.google.com/"; }
set { base.Referer = value; }
}
public override string RegexMatch
{
get { return @"<div id="result_box" dir="\""ltr\""">((?:.|\n)*?)</div>"; }
set { base.RegexMatch = value; }
}
public override string Name
{
get { return "Google"; }
}
public override bool CanTranslate( TranslationMode mode )
{
var sourceFound = s_SourceLanguages.Count(s => s.StartsWith(mode.SourceLanguage, StringComparison.InvariantCultureIgnoreCase)) > 0;
var targetFound = s_TargetLanguages.Count(s => s.StartsWith(mode.TargetLanguage, StringComparison.InvariantCultureIgnoreCase)) > 0;
if (sourceFound && targetFound)
{
return true;
}
else
{
Trace.TraceWarning("Translation mode is not supported: " + mode);
return false;
}
}
protected override string GetPostSourceData( TranslationMode mode, string text )
{
return string.Format("hl={0}&ie=UTF8&text={1}&sl={2}&tl={3}",
mode.SourceLanguage, HttpUtility.UrlEncode(text), mode.SourceLanguage, mode.TargetLanguage);
}
internal class GoogleTranslationMode : TranslationMode
{
public override string SourceLanguage
{
get
{
if (!base.SourceLanguage.StartsWith("zh"))
{
return base.SourceLanguage.Substring(0, 2);
}
else
{
return base.SourceLanguage;
}
}
set { base.SourceLanguage = value; }
}
public override string TargetLanguage
{
get
{
if (!base.TargetLanguage.StartsWith("zh"))
{
return base.TargetLanguage.Substring(0, 2);
}
else
{
return base.TargetLanguage;
}
}
set { base.TargetLanguage = value; }
}
public override string ToString()
{
return SourceLanguage + " to " + TargetLanguage;
}
}
}
}
|
|
|
|
|
|
|
You are the man!
Code works perfectly (after playing with regexes!).
I Love You.
|
|
|
|
|
| webservice for Translating
angelsherin | 3:24 29 Jun '08 |
|
|
Hi friends,
I am working in windows applications.My current task is:
I have a windows form which contains a textBox and Button.
i have to input a text thro textbox in english(Via TextBox)ex:"Open" when i click the button, it calls a language translation webservice and it should return the corresponding text in German.Here in our case it should return the corresponding text of "Open" in German.
I need sample code my friendsand the webservice URL[Language translation web service]
NOte: For eaxample purpose only i have given English to german..
In real time it may be any language(X) to any language(Y)
So i need webservice url which converts any langusge to any language
Expecting help from U friends...Thanks in advance...Sherin
|
|
|
|
|
| Translating webservice
angelsherin | 3:24 29 Jun '08 |
|
|
Hi friends,
I am working in windows applications.My current task is:
I have a windows form which contains a textBox and Button.
i have to input a text thro textbox in english(Via TextBox)ex:"Open" when i click the button, it calls a language translation webservice and it should return the corresponding text in German.Here in our case it should return the corresponding text of "Open" in German.
I need sample code my friendsand the webservice URL[Language translation web service]
NOte: For eaxample purpose only i have given English to german..
In real time it may be any language(X) to any language(Y)
So i need webservice url which converts any langusge to any language
Expecting help from U friends...Thanks in advance...Sherin
|
|
|
|
|
| Not working
Eugen Wiebe | 1:03 4 Jun '08 |
|
|
The web service is not working because the translation provider has changed his interface. Who knows an other way to online translate?
|
|
|
|
|
| No Babelfish web service
Sandeepan | 2:12 17 Apr '08 |
|
|
I guess the babelfish webservice is not available anymore...
Rather this webservice link can be used for free
Translate Service
[http://www.webservicex.net/TranslateService.asmx">http://www.webservicex.net/TranslateService.asmx]
Sandeepan
|
|
|
|
|
|
|
Why the BabelFish WebService is not available?
|
|
|
|
|
|
|
they have shutdown the service - you have to $ if you want to use their service
amok
|
|
|
|
|
 | no "test" button appear
Gabriyel | 18:58 11 Jun '07 |
|
|
hi,
when i tried accessing the translate.asmx file, there's no "test" button on the page. instead, a statement saying "The following operations are supported..." looks like a standard message to indicate the page as a webservice.
how can i view the page with the "Test" button and textboxes for me to fill in the sample text?
thanks.
|
|
|
|
|
| Big Chunk of Content with HTML Tags
pbansal | 2:45 12 Mar '07 |
|
|
Hello again,
I need to convert big chunk of content with a lot of html tags. I do not want to translate the html tags but only the content within the html tags.
Could anyone suggest me the right way..
Thanks and Regards
Pawan
|
|
|
|
|
| Unable to connect to the remote server
pbansal | 1:46 12 Mar '07 |
|
|
Even after adding the required patch in web.config, this does not seem to work for me. Please if anyone can upload the running project again using VB.Net for Google Translator.
Erro is coming as
"""The underlying connection was closed: Unable to connect to the remote server.ERROR3:There was a problem connecting to the Google server
"""
with thanks
|
|
|
|
|
| URGENT HELP REQUIRED
HARISHRAM | 4:39 13 Dec '06 |
|
|
I AM ENCOUNTERING THIS PROBLEM SAYING THAT IT CANNOT FIND THE TRANSLATE.CSPROJ IN LOCAL HOST. CAN ANYONE HELP ME WITH THAT. I DESPERATELY NEED THIS TO BE SOLVED. THANKS
|
|
|
|
|
| There was a problem connecting to the Babelfish server
bijulsoni | 3:09 14 Nov '06 |
|
|
hello
i am trying to implement Windows program using your web service.But the problem is i got one exception on
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
The exception string is
"There was a problem connecting to the Babelfish server"
we r using broadband connections so there is no scope for request timeout and i have checked using URL that BabeFish server is up and running..
can you pls help me
|
|
|
|
|
|
|
Try adding this segment to Web.comfig file. It worked for me.
<configuration>
<system.net>
<settings>
<httpWebRequest useUnsafeHeaderParsing="true" />
</settings>
</system.net>
...
</configuration>
It will instruct your app to parse malformed headers. You can find more on this at http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=21106&SiteID=1&PageID=0
Dimitar
|
|
|
|
|
|
|
Thank You very much
It worked for me..
Thanks Again....
|
|
|
|
|
|
|
I tried to put that in my app.config file, but it doesn't help. I get the same "There was a problem connecting to the Babelfish server" error.
I am doing this in a Windows Application. Is there something else I need to do for that?
Thanks for your time and help!
Brenda
|
|
|
|
|
| Incorrect content-length header
Bret Mulvey | 6:42 20 Oct '06 |
|
|
This code sets the content-length header of the request to the number of characters in the string. Instead, it should be the number of bytes in the request body (the length of the "bytes" array). It works for English because all the characters get encoded as one byte, but may not work for other languages if Babel Fish is paying attention to this header. It may be ignoring this header in which case it doesn't matter, but this should probably be fixed at some point.
|
|
|
|
|
| Using VC++ ???
Dingos Pingo | 19:04 10 Aug '06 |
|
|
Hi!
How can I use the same feature using VC++ development environment.
Regards,
Milind
|
|
|
|
|
|
|
See this[^] article.
/ravi
|
|
|
|
|
|
|
Last Updated 19 Apr 2004 |
Advertise |
Privacy |
Terms of Use |
Copyright ©
CodeProject, 1999-2010
