Click here to Skip to main content
Click here to Skip to main content
Add your own
alternative version
Go to top

Wave: a Standard conformant C++ preprocessor library

, 10 Jan 2004
Describes a free and fully Standard conformant C++ preprocessor library
wave_preprocessor_demo.zip
wave.exe
wave_preprocessor_demo1.zip
wave.exe
wave_preprocessor_src.zip
wave
doc
theme
bkd.gif
bkd2.gif
bullet.gif
l_arr.gif
l_arr_disabled.gif
r_arr.gif
r_arr_disabled.gif
uc.gif
u_arr.gif
wave.gif
test
boost-build.jam
cpp_tokens
Jamfile.v2
list_includes
project-root.jam
wave
Jamfile
Jamfile.v2
runtests.sh
test_files
wave
cpplexer
re2clex
cpp.re
test
lextest.in
lextest.output
run_tests.sh
test_lexer.in
test_lexer.output
slex
test
lextest.in
lextest.re2c.output
lextest.slex.output
run_tests.sh
grammars
idllexer
re2clex
idl.re
util
wave_preprocessor_src1.zip
bkd.gif
bkd2.gif
bullet.gif
l_arr.gif
l_arr_disabled.gif
r_arr.gif
r_arr_disabled.gif
u_arr.gif
uc.gif
wave.gif
boost-build.jam
Jamfile.v2
project-root.jam
Jamfile
Jamfile.v2
runtests.sh
cpp.re
lextest.in
lextest.output
run_tests.sh
test_lexer.in
test_lexer.output
lextest.in
lextest.re2c.output
lextest.slex.output
run_tests.sh
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>The Token Type</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link rel="stylesheet" href="theme/style.css" type="text/css">
</head>
<body text="#000000" background="theme/bkd.gif">
<table width="100%" border="0" cellspacing="2" background="theme/bkd2.gif">
  <tr> 
    <td width="21"> <h1></h1></td>
    <td width="885"> <font face="Verdana, Arial, Helvetica, sans-serif"><b><font size="6">The 
      Token Type</font></b></font></td>
    <td width="96"><a href="http://spirit.sf.net"><img src="theme/wave.gif" width="93" height="68" align="right" border="0"></a></td>
  </tr>
</table>
<br>
<table border="0">
  <tr> 
    <td width="10"></td>
    <td width="30"><a href="index.html"><img src="theme/u_arr.gif" border="0"></a></td>
    <td width="30"><a href="class_reference_lexer.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td>
    <td width="30"><a href="token_ids.html"><img src="theme/r_arr.gif" border="0"></a></td>
  </tr>
</table>
<blockquote>
  <p><a href="class_reference_tokentype.html#introduction">Introduction</a><br>
    <a href="class_reference_tokentype.html#header_synopsis">Header 'wave/context.hpp' 
    synopsis</a><br>
    <a href="class_reference_tokentype.html#template_parameters">Template parameters</a><br>
    <a href="class_reference_tokentype.html#member_functions">Member functions</a></p>
</blockquote>
<h2><a name="Introduction"></a>Introduction</h2>
<p>The token type in <tt>Wave</tt> is the main carrier of information. It is returned 
  by dereferencing the main iterator. The tokens are originally generated by the 
  lexer (&quot;An entity that lexically transforms the subject of parsing to a 
  sequence of objects (called tokens) more suitable for subsequent parsing.&quot;). 
  The Wave library contains two different, interchangable C++ lexers, which may 
  be used as a starting point during developing your own application. The lexer 
  generated tokens are transformed by the preprocessing engine (macro expansion, 
  include file processing etc.) and after this returned to the user of the <tt>Wave</tt> 
  library.</p>
<p>You can use arbitrary token types in conjunction with your own lexer as long 
  as these implement some required interface. The required token type interface 
  is described below and is implemented by the <tt>wave::cpplexer::lex_token</tt> 
  template, the required lexer interface is described <a href="class_reference_lexer.html">here</a>. 
</p>
<p>In the following is described the token type predefined inside the <tt>Wave</tt> 
  library, which is used in conjunction with the two predefined C++ lexers contained 
  in the <tt>Wave</tt> library too. If you need to use your own token type, it 
  is required to implement the interafce described below.</p>
<h2><b><a name="header_synopsis"></a>Header <a href="http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/spirit/spirit/wave/wave/cpplexer/cpp_lex_token.hpp?rev=HEAD&content-type=text/vnd.viewcvs-markup">wave/cpplexer/cpp_lex_token.hpp</a> 
  synopsis</b></h2>
<pre><span class="keyword"><code>namespace</code></span><code> wave {
<span class="keyword">namespace</span> cpplexer {

    <span class="keyword">template</span> &lt;<span class="keyword">typename</span> PositionT&gt;
    <span class="keyword">class</span> lex_token 
    {
    <span class="keyword">public</span>:
        <span class="keyword">typedef</span> <span class="keyword">std::basic_string</span><char> string_t;
        <span class="keyword">typedef</span> PositionT         position_t;
    
        <a href="class_reference_tokentype.html#constructor">lex_token</a>();
        <a href="class_reference_tokentype.html#constructor">lex_token</a>(token_id id, string_t <span class="keyword">const</span> &value, 
            PositionT <span class="keyword">const</span> &pos);

    // accessors
        <a href="class_reference_tokentype.html#operator_tokenid">operator token_id</a>() <span class="keyword">const</span>;
        string_t const &amp;<a href="class_reference_tokentype.html#get_value">get_value</a>() <span class="keyword">const</span>;
        position_t const &amp;<a href="class_reference_tokentype.html#get_position">get_position</a>() <span class="keyword">const</span>;
        void <a href="class_reference_tokentype.html#set_token_id">set_token_id</a> (token_id id);
        void <a href="class_reference_tokentype.html#set_value">set_value</a> (string_t <span class="keyword">const</span> &amp;newval);
        void <a href="class_reference_tokentype.html#set_position">set_position</a> (position_t <span class="keyword">const</span> &amp;pos);
    };

}   <span class="comment">// namespace cpplexer</span>
}   <span class="comment">// namespace wave</span>
</code></pre>
<h2><b><a name="template_parameters" id="template_parameters"></a>Template parameters</b></h2>
<p>The predefined token type uses a template parameter <tt>PositionT</tt>, which 
  allows to specify the type to be used to carry the file position information 
  contained inside the generated tokens. This type should contain at least the 
  filename, the line number and the column number of the position, where the token 
  was recognized. It defaults to a simple file_position template class described 
  <a href="class_reference_fileposition.html">here</a>. </p>
<h2><b><a name="member_functions"></a>Member functions</b></h2>
<p><b><a name="constructor" id="constructor"></a>Constructors</b></p>
<pre>    lex_token();

    lex_token(token_id id, 
        string_t <span class="keyword">const</span> &amp;value, 
        PositionT <span class="keyword">const</span> &amp;pos);</pre>
<blockquote> 
  <p>The first (default) constructor is for generating an end of stream token, 
    which is used for indicating the end of the underlying input stream.</p>
  <p>The second constructor initializes the newly created token object with its 
    token id (for a list of valid token id's please look <a href="token_ids.html">here</a>), 
    the string representation of its value and the file position, describing the 
    position inside the input stream , where this token was recognized.</p>
</blockquote>
<h2>Accessor functions</h2>
<p><a name="operator_tokenid"></a><b>operator token_id</b></p>
<pre>    <span class="keyword">operator</span> token_id() <span class="keyword">const</span>;</pre>
<blockquote> 
  <p>Allows the access to the token id of the token. This access function is implemented 
    as an implicit conversion function to allow the operation of <tt>Spirit</tt> 
    parsers directly on top of a token stream. The possible token id's are described 
    <a href="token_ids.html">here</a>. </p>
  <p>This function does not throw in any case.</p>
</blockquote>
<p><a name="get_value"></a><b>get_value</b></p>
<pre>    string_t <span class="keyword">const</span> &amp;get_value() <span class="keyword">const</span>;</pre>
<blockquote> 
  <p>Returns the value of the token, as it was recognized in the input stream. 
    Even for constant tokens (as keywords or operators etc.) the returned value 
    reflects the character sequence as found in the input stream. </p>
  <p>This function does not throw in any case.</p>
</blockquote>
<p><b><a name="get_position"></a>get_position</b></p>
<pre>    PositionT <span class="keyword">const</span> &amp;get_position() <span class="keyword">const</span>;</pre>
<blockquote> 
  <p>Returns the position of the token in the input stream, where it was recognized. 
    The position contains information about the filename, the line number and 
    the column number of the token. By default the <tt>Wave</tt> library uses 
    a file_position template for this purpose, which is described in more detail 
    <a href="class_reference_fileposition.html">here</a>.</p>
  <p>This function does not throw in any case.</p>
</blockquote>
<p><a name="set_token_id"></a><b>set_token_id</b></p>
<pre>    void set_token_id(token_id id);</pre>
<blockquote> 
  <p>Changes the token id of the token to the new value. The possible token id's 
    are described <a href="token_ids.html">here</a>. Propably this function is 
    of little value for the library user, but it is required under certain circumstances 
    for correct operation of the preprocessing engine.</p>
  <p>This function does not throw in any case.</p>
</blockquote>
<p><a name="set_value"></a><b>set_value</b></p>
<pre>    void set_value(string_t <span class="keyword">const</span> &amp;newval);</pre>
<blockquote> 
  <p>Changes the value stored inside the token to the new value. Propably this 
    function is of little value for the library user, but it is required under 
    certain circumstances for correct operation of the preprocessing engine.</p>
</blockquote>
<p><a name="set_position"></a><b>set_position</b></p>
<pre>    void set_position(PositionT <span class="keyword">const</span> &amp;newpos);</pre>
<blockquote>
  <p>Changes the position stored inside the token to the new value. This is used 
    for instance for implementing the functionality required for to implement 
    the <tt>#line</tt> directive.</p>
</blockquote>
<table border="0">
  <tr> 
    <td width="10"></td>
    <td width="30"><a href="index.html"><img src="theme/u_arr.gif" border="0"></a></td>
    <td width="30"><a href="class_reference_lexer.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td>
    <td width="30"><a href="token_ids.html"><img src="theme/r_arr.gif" border="0"></a></td>
  </tr>
</table>
<hr size="1">
<p class="copyright">Copyright &copy; 2003-2004 Hartmut Kaiser<br>
  <br>
  <font size="2">Use, modification and distribution is subject to the Boost Software 
  License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) 
  </font> </p>
<span class="updated">Last updated: 
  <!-- #BeginDate format:fcAm1m -->Monday, January 5, 2004  14:57<!-- #EndDate -->
  </span></body>
</html>

By viewing downloads associated with this article you agree to the Terms of Service and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Hartmut Kaiser

United States United States
Actively involved in Boost and the development of the Spirit parser construction framework.

| Advertise | Privacy | Mobile
Web01 | 2.8.140921.1 | Last Updated 11 Jan 2004
Article Copyright 2003 by Hartmut Kaiser
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid