Click here to Skip to main content
Click here to Skip to main content
Add your own
alternative version
Go to top

Wave: a Standard conformant C++ preprocessor library

, 10 Jan 2004
Describes a free and fully Standard conformant C++ preprocessor library
wave_preprocessor_demo.zip
wave.exe
wave_preprocessor_demo1.zip
wave.exe
wave_preprocessor_src.zip
wave
doc
theme
bkd.gif
bkd2.gif
bullet.gif
l_arr.gif
l_arr_disabled.gif
r_arr.gif
r_arr_disabled.gif
uc.gif
u_arr.gif
wave.gif
test
boost-build.jam
cpp_tokens
Jamfile.v2
list_includes
project-root.jam
wave
Jamfile
Jamfile.v2
runtests.sh
test_files
wave
cpplexer
re2clex
cpp.re
test
lextest.in
lextest.output
run_tests.sh
test_lexer.in
test_lexer.output
slex
test
lextest.in
lextest.re2c.output
lextest.slex.output
run_tests.sh
grammars
idllexer
re2clex
idl.re
util
wave_preprocessor_src1.zip
bkd.gif
bkd2.gif
bullet.gif
l_arr.gif
l_arr_disabled.gif
r_arr.gif
r_arr_disabled.gif
u_arr.gif
uc.gif
wave.gif
boost-build.jam
Jamfile.v2
project-root.jam
Jamfile
Jamfile.v2
runtests.sh
cpp.re
lextest.in
lextest.output
run_tests.sh
test_lexer.in
test_lexer.output
lextest.in
lextest.re2c.output
lextest.slex.output
run_tests.sh
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>The Lexer Interface</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link href="theme/style.css" rel="stylesheet" type="text/css">
</head>

<body>
<table width="100%" border="0" cellspacing="2" background="theme/bkd2.gif">
  <tr> 
    <td width="21"> <h1></h1></td>
    <td width="885"> <font face="Verdana, Arial, Helvetica, sans-serif"><b><font size="6">The 
      Lexer Interface</font></b></font></td>
    <td width="96"><a href="http://spirit.sf.net"><img src="theme/wave.gif" width="93" height="68" align="right" border="0"></a></td>
  </tr>
</table>
<br>
<table border="0">
  <tr> 
    <td width="10"></td>
    <td width="30"><a href="index.html"><img src="theme/u_arr.gif" border="0"></a></td>
    <td width="30"><a href="class_reference_tracepolicy.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td>
    <td width="30"><a href="class_reference_tokentype.html"><img src="theme/r_arr.gif" border="0"></a></td>
  </tr>
</table>
<blockquote>
  <p><a href="class_reference_lexer.html#introduction">Introduction</a><br>
    <a href="class_reference_lexer.html#header_synopsis">Header 'wave/cpplexer/cpp_lexer_interface.hpp' 
    synopsis</a><br>
    <a href="class_reference_lexer.html#template_parameters">Template parameters</a><br>
    <a href="class_reference_lexer.html#member_functions">Member functions</a></p>
</blockquote>
<h2><b><a name="introduction"></a>Introduction</b></h2>
<p>The <tt>wave::cpplexer::lex_interface</tt> template is designed to be used 
  as a base class for any C++ lexer, which should be plugged in into the Wave 
  preprocessor engine. It is an abstract base class, which allows to abstract 
  the lexer itself from the other parts of the library. Additionally it contains 
  a static functions, through which a new instance of the lexer object should 
  be instantiated. For this to function correctly you need to supply an implementation 
  for the <tt>new_lexer_gen</tt> template, which actually is used to generate 
  a new instance of your lexer object. This lexer object is expected to have a 
  constructor taking three parameters: the begin and end iterators of the underlying 
  input stream to analyse and the file_position of the point in the input stream, 
  which corresponds to the first of the given iterators.</p>
<h2><a name="header_synopsis"></a>Header <a href="http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/spirit/spirit/wave/wave/cpplexer/cpp_lex_interface.hpp?rev=HEAD&content-type=text/vnd.viewcvs-markup">wave/cpplexer/cpp_lexer_interface.hpp</a> 
  synopsis</h2>
<pre><span class="keyword">namespace</span> wave {
<span class="keyword">namespace</span> cpplexer {
 
    <span class="keyword">template</span> &lt;<span class="keyword">typename</span> IteratorT, <span class="keyword">typename</span> PositionT&gt;
    <span class="keyword">struct</span> new_lexer_gen
    {
        <span class="keyword">static</span> lex_input_interface&lt;lex_token&lt;PositionT&gt; &gt; *
        <a href="class_reference_lexer.html#new_lexer_gen_new_lexer">new_lexer</a>(IteratorT <span class="keyword">const</span> &amp;first, IteratorT <span class="keyword">const</span> &amp;last, 
            PositionT <span class="keyword">const</span> &amp;pos);<br>    };

    <span class="keyword">template</span> &lt;<span class="keyword">typename</span> TokenT&gt;
    <span class="keyword">struct</span> lex_input_interface 
    {
        <span class="keyword">typedef typename</span> TokenT::position_t position_t;
    
        <span class="keyword">virtual</span> TokenT <a href="class_reference_lexer.html#get">get</a>()<span class="keyword"> = 0</span>;
        <span class="keyword">virtual void</span> <a href="class_reference_lexer.html#set_position">set_position</a>(position_t const &pos)<span class="keyword"> = 0</span>;

        <span class="keyword">virtual</span> ~<a href="class_reference_lexer.html#destructor">lex_input_interface</a>() {}

    <span class="comment">//  The new_lexer function allows the opaque generation of </span>
    <span class="comment">//  a new lexer object. It is coupled to the token type </span>
    <span class="comment">//  to allow to distinguish different lexer/token 
    //  configurations at compile time.</span>
        <span class="keyword">template</span> &lt;<span class="keyword">typename</span> IteratorT&gt;
        <span class="keyword">static</span> lex_input_interface *
        new_lexer(IteratorT <span class="keyword">const</span> &amp;first, IteratorT <span class="keyword">const</span> &amp;last, 
            position_t const &amp;pos)
        { 
            <span class="keyword">return</span> new_lexer_gen&lt;IteratorT, position_t&gt;::
                new_lexer (first, last, pos); 
        }
    }
};

}   <span class="comment">// namespace cpplexer</span>
}   <span class="comment">// namespace wave</span>

</pre>
<h2><a name="template_parameters"></a>Template parameters</h2>
<p>The <tt>lex_input_interface</tt> itself needs the token type to be supplied 
  as the only template parameter. It specifies the token type to be returned by 
  the actual C++ lexer. The predefined C++ lexers inside the <tt>Wave</tt> library 
  use the lex_token template (described <a href="class_reference_tokentype.html">here</a>) 
  for this purpose.</p>
<p>The <tt>new_lexer_gen</tt> structure needs to be implemented besides the actual 
  lexer object. It takes two template parameters: the iterator type of the underlying 
  input stream to use and the type of the <tt>file_position</tt> struct to be 
  used by the C++ lexer. If the second parameter is omitted, it should default 
  to the <tt>PositionT</tt> template parameter used by the token type.</p>
<h2><a name="member_functions"></a>Member functions</h2>
<p><a name="destructor"></a><b>destructor</b></p>
<pre>    <span class="keyword">virtual</span> ~lex_input_interface();</pre>
<blockquote>
  <p>The destructor is defined as <tt>virtual</tt> to garant the correct destruction 
    of the derived lexers. This is very important to avoid memory leaks.</p>
</blockquote>
<p><b><a name="get"></a>get</b></p>
<pre><span class="keyword">    virtual</span> TokenT get();</pre>
<blockquote>
  <p>This function is called, whenever the next token from the input stream is 
    needed. It should return this token, if it is available otherwise the <tt>T_EOF</tt> 
    token (see <a href="token_ids.html">The Token Identifiers</a>), signaling 
    the end of the underlying input stream.</p>
</blockquote>
<p><b><a name="set_position"></a>set_position</b></p>
<pre><span class="keyword">    virtual void</span> set_position(
        position_t const &pos)<span class="keyword">;</span></pre>
<blockquote>
  <p>This function is called, whenever the current position of the input stream 
    has to be changed. Note, that this function should <b>not</b> move the iterators 
    of the input stream, it should instead store the new position and take it 
    as the reference point for all subsequent generated tokens. For instance this 
    function is called , whenever a valid <tt>#line</tt> directive is recognized 
    in the input stream. </p>
</blockquote>
<p><a name="new_lexer_gen_new_lexer"></a><b>new_lexer_gen::new_lexer</b></p>
<pre><span class="keyword">    static</span> lex_input_interface&lt;TokenT&gt; *
    new_lexer(IteratorT <span class="keyword">const</span> &amp;first, IteratorT <span class="keyword">const</span> &amp;last, 
            PositionT <span class="keyword">const</span> &amp;pos);</pre>
<blockquote>
  <p>&nbsp;</p>
</blockquote>
<table border="0">
  <tr> 
    <td width="10"></td>
    <td width="30"><a href="index.html"><img src="theme/u_arr.gif" border="0"></a></td>
    <td width="30"><a href="class_reference_tracepolicy.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td>
    <td width="30"><a href="class_reference_tokentype.html"><img src="theme/r_arr.gif" border="0"></a></td>
  </tr>
</table>
<hr size="1">
<p class="copyright">Copyright &copy; 2003-2004 Hartmut Kaiser<br>
  <br>
  <font size="2">Use, modification and distribution is subject to the Boost Software 
  License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) 
  </font> </p>
<span class="updated">Last updated: 
  <!-- #BeginDate format:fcAm1m -->Monday, January 5, 2004  14:57<!-- #EndDate -->
  </span></body>
</html>

By viewing downloads associated with this article you agree to the Terms of Service and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Hartmut Kaiser

United States United States
Actively involved in Boost and the development of the Spirit parser construction framework.

| Advertise | Privacy | Mobile
Web02 | 2.8.140916.1 | Last Updated 11 Jan 2004
Article Copyright 2003 by Hartmut Kaiser
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid