Detect a written text's language

Carsten Zeumer

Rate me:

4.96/5 (75 votes)

21 Oct 2009CPOL6 min read

155K

7.7K

114

An article on how to detect the language of a written text.

babel.src.zip
- DialogueMaster.Babel
  - app.config
  - App.ico
  - AssemblyInfo.cs
  - BabelInstaller.cs
  - BabelInstaller.resx
  - BabelModel.cs
  - Class1.cs
  - DialogueMaster.Babel.csproj
  - DialogueMaster.Babel.sln
  - DialogueMaster.cs
  - models
    - all.Model
  - TokenStats.cs
  - TokenTable.cs
  - TokenVoter.cs
  - WinSample
    - DemoForm.cs
    - DemoForm.Designer.cs
    - DemoForm.resx
    - Program.cs
    - Properties
    - WinSample.csproj
babel.winsample.zip
- Babel.WinSample
  - DialogueMaster.Babel.dll
  - WinSample.exe


Microsoft Visual Studio Solution File, Format Version 10.00
# Visual Studio 2008
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DialogueMaster.Babel", "DialogueMaster.Babel.csproj", "{3825704F-160A-4EE1-928C-A980D6C96224}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Test", "Test\Test\Test.csproj", "{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "lib", "lib", "{E55BC23F-236A-4CBE-BAC4-AE7106B4D29F}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DialogueMaster.Core.Interfaces", "..\DialogueMaster.Core.Interfaces\DialogueMaster.Core.Interfaces.csproj", "{DA3C4927-5041-4B1A-BB83-AE5343205D84}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DialogueMaster.Collections", "..\DialogueMaster.Collections\DialogueMaster.Collections.csproj", "{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "WinSample", "WinSample\WinSample.csproj", "{63D08917-B62B-4181-A7AF-2B4E194DEB7C}"
EndProject
Global
	GlobalSection(SolutionConfigurationPlatforms) = preSolution
		Debug|Any CPU = Debug|Any CPU
		Debug|x64 = Debug|x64
		Debug|x86 = Debug|x86
		DialogueMaster.Net test|Any CPU = DialogueMaster.Net test|Any CPU
		DialogueMaster.Net test|x64 = DialogueMaster.Net test|x64
		DialogueMaster.Net test|x86 = DialogueMaster.Net test|x86
		Release|Any CPU = Release|Any CPU
		Release|x64 = Release|x64
		Release|x86 = Release|x86
	EndGlobalSection
	GlobalSection(ProjectConfigurationPlatforms) = postSolution
		{3825704F-160A-4EE1-928C-A980D6C96224}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
		{3825704F-160A-4EE1-928C-A980D6C96224}.Debug|Any CPU.Build.0 = Debug|Any CPU
		{3825704F-160A-4EE1-928C-A980D6C96224}.Debug|x64.ActiveCfg = Debug|x64
		{3825704F-160A-4EE1-928C-A980D6C96224}.Debug|x64.Build.0 = Debug|x64
		{3825704F-160A-4EE1-928C-A980D6C96224}.Debug|x86.ActiveCfg = Debug|x86
		{3825704F-160A-4EE1-928C-A980D6C96224}.Debug|x86.Build.0 = Debug|x86
		{3825704F-160A-4EE1-928C-A980D6C96224}.DialogueMaster.Net test|Any CPU.ActiveCfg = DialogueMaster.Net test|Any CPU
		{3825704F-160A-4EE1-928C-A980D6C96224}.DialogueMaster.Net test|Any CPU.Build.0 = DialogueMaster.Net test|Any CPU
		{3825704F-160A-4EE1-928C-A980D6C96224}.DialogueMaster.Net test|x64.ActiveCfg = DialogueMaster.Net test|x64
		{3825704F-160A-4EE1-928C-A980D6C96224}.DialogueMaster.Net test|x64.Build.0 = DialogueMaster.Net test|x64
		{3825704F-160A-4EE1-928C-A980D6C96224}.DialogueMaster.Net test|x86.ActiveCfg = DialogueMaster.Net test|x86
		{3825704F-160A-4EE1-928C-A980D6C96224}.DialogueMaster.Net test|x86.Build.0 = DialogueMaster.Net test|x86
		{3825704F-160A-4EE1-928C-A980D6C96224}.Release|Any CPU.ActiveCfg = Release|Any CPU
		{3825704F-160A-4EE1-928C-A980D6C96224}.Release|Any CPU.Build.0 = Release|Any CPU
		{3825704F-160A-4EE1-928C-A980D6C96224}.Release|x64.ActiveCfg = Release|x64
		{3825704F-160A-4EE1-928C-A980D6C96224}.Release|x64.Build.0 = Release|x64
		{3825704F-160A-4EE1-928C-A980D6C96224}.Release|x86.ActiveCfg = Release|x86
		{3825704F-160A-4EE1-928C-A980D6C96224}.Release|x86.Build.0 = Release|x86
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Debug|Any CPU.Build.0 = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Debug|x64.ActiveCfg = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Debug|x86.ActiveCfg = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.DialogueMaster.Net test|Any CPU.ActiveCfg = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.DialogueMaster.Net test|Any CPU.Build.0 = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.DialogueMaster.Net test|x64.ActiveCfg = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.DialogueMaster.Net test|x86.ActiveCfg = Debug|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Release|Any CPU.ActiveCfg = Release|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Release|Any CPU.Build.0 = Release|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Release|x64.ActiveCfg = Release|Any CPU
		{53198C48-56A0-49E8-8F3A-8A1EEB188AE6}.Release|x86.ActiveCfg = Release|Any CPU
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Debug|Any CPU.Build.0 = Debug|Any CPU
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Debug|x64.ActiveCfg = Debug|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Debug|x64.Build.0 = Debug|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Debug|x86.ActiveCfg = Debug|x86
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Debug|x86.Build.0 = Debug|x86
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.DialogueMaster.Net test|Any CPU.ActiveCfg = Debug|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.DialogueMaster.Net test|x64.ActiveCfg = Debug|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.DialogueMaster.Net test|x64.Build.0 = Debug|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.DialogueMaster.Net test|x86.ActiveCfg = Debug|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Release|Any CPU.ActiveCfg = Release|Any CPU
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Release|Any CPU.Build.0 = Release|Any CPU
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Release|x64.ActiveCfg = Release|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Release|x64.Build.0 = Release|x64
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Release|x86.ActiveCfg = Release|x86
		{DA3C4927-5041-4B1A-BB83-AE5343205D84}.Release|x86.Build.0 = Release|x86
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Debug|Any CPU.Build.0 = Debug|Any CPU
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Debug|x64.ActiveCfg = Debug|x64
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Debug|x64.Build.0 = Debug|x64
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Debug|x86.ActiveCfg = Debug|x86
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Debug|x86.Build.0 = Debug|x86
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.DialogueMaster.Net test|Any CPU.ActiveCfg = DialogueMaster.Net test|Any CPU
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.DialogueMaster.Net test|Any CPU.Build.0 = DialogueMaster.Net test|Any CPU
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.DialogueMaster.Net test|x64.ActiveCfg = DialogueMaster.Net test|x64
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.DialogueMaster.Net test|x64.Build.0 = DialogueMaster.Net test|x64
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.DialogueMaster.Net test|x86.ActiveCfg = DialogueMaster.Net test|x86
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.DialogueMaster.Net test|x86.Build.0 = DialogueMaster.Net test|x86
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Release|Any CPU.ActiveCfg = Release|Any CPU
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Release|Any CPU.Build.0 = Release|Any CPU
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Release|x64.ActiveCfg = Release|x64
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Release|x64.Build.0 = Release|x64
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Release|x86.ActiveCfg = Release|x86
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A}.Release|x86.Build.0 = Release|x86
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Debug|Any CPU.Build.0 = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Debug|x64.ActiveCfg = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Debug|x86.ActiveCfg = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.DialogueMaster.Net test|Any CPU.ActiveCfg = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.DialogueMaster.Net test|Any CPU.Build.0 = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.DialogueMaster.Net test|x64.ActiveCfg = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.DialogueMaster.Net test|x86.ActiveCfg = Debug|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Release|Any CPU.ActiveCfg = Release|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Release|Any CPU.Build.0 = Release|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Release|x64.ActiveCfg = Release|Any CPU
		{63D08917-B62B-4181-A7AF-2B4E194DEB7C}.Release|x86.ActiveCfg = Release|Any CPU
	EndGlobalSection
	GlobalSection(SolutionProperties) = preSolution
		HideSolutionNode = FALSE
	EndGlobalSection
	GlobalSection(NestedProjects) = preSolution
		{DA3C4927-5041-4B1A-BB83-AE5343205D84} = {E55BC23F-236A-4CBE-BAC4-AE7106B4D29F}
		{C3F87549-6E3E-4050-B6E7-4F0ECCD2E84A} = {E55BC23F-236A-4CBE-BAC4-AE7106B4D29F}
	EndGlobalSection
EndGlobal

By viewing downloads associated with this article you agree to the Terms of Service and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By

Carsten Zeumer

Software Developer (Senior)

Germany

Carsten started programming Basic and Assembler back in the 80’s when he got his first C64. After switching to a x86 based system he started programming in Pascal and C. He started Windows programming with the arrival of Windows 3.0. After working for various internet companies developing a linguistic text analysis and classification software for 25hours communications he is now working as a contractor.

Carsten lives in Hamburg, Germany with his wife and five children.

Detect a written text's language

License

Comments and Discussions