empy

Summary

   A templating system for Python.

Overview

   EmPy is a system for embedding Python expressions and statements in template text; it
   takes an EmPy source file, processes it, and produces output. This is accomplished via
   expansions, which are special signals to the EmPy system and are set off by a special
   prefix (by default the at sign, @). EmPy can expand arbitrary Python expressions and
   statements in this way, as well as a variety of special forms. Textual data not explicitly
   delimited in this way is sent unaffected to the output, allowing Python to be used in
   effect as a markup language. Also supported are "hook" callbacks, recording and playback
   via diversions, and dynamic, chainable filters. The system is highly configurable via
   command line options and embedded commands.

   Expressions are embedded in text with the @(...) notation; variations include conditional
   expressions with @(...?...:...) and the ability to handle thrown exceptions with
   @(...$...). As a shortcut, simple variables and expressions can be abbreviated as
   @variable, @object.attribute, @function(arguments), @sequence [2][index]
   , and combinations. Full-fledged statements are embedded with @{...}. Forms of
   conditional, repeated, and recallable expansion are available via @[...]. A @ followed by
   a whitespace character (including a newline) expands to nothing, allowing string
   concatenations and line continuations. Comments are indicated with @# and consume the rest
   of the line, up to and including the trailing newline. @% indicate "significators," which
   are special forms of variable assignment intended to specify per-file identification
   information in a format which is easy to parse externally. Escape sequences analogous to
   those in C can be specified with @\..., and finally a @@ sequence expands to a single
   literal at sign.

Getting the software

   The current version of empy is 2.3.

   The latest version of the software is available in a tarball here: [3]
   http://www.alcyone.com/pyos/empy/empy-latest.tar.gz.

   The official URL for this Web site is [4]http://www.alcyone.com/pyos/empy/.

Requirements

   EmPy should work with any version of Python from 1.5.x onward. It has been tested with all
   major versions of CPython from 1.5 up, and Jython from 2.0 up. The included test script is
   intended to run on UNIX-like systems with a Bourne shell.

License

   This code is released under the [5]GPL.

Mailing lists

   There are two EmPy related mailing lists available. The first is a receive-only, very low
   volume list for important announcements (including releases). To subscribe, send an email
   to [6]empy-announce-list-subscribe@alcyone.com.

   The second is a general discussion list for topics related to EmPy, and is open for
   everyone to contribute; announcements related to EmPy will also be made on this list. The
   author of EmPy (and any future developers) will also be on the list, so it can be used not
   only to discuss EmPy features with other users, but also to ask questions of the
   author(s). To subscribe, send an email to [7]empy-list-subscribe@alcyone.com.

Basics

   EmPy is intended for embedding Python code in otherwise unprocessed text. Source files are
   processed, and the results are written to an output file. Normal text is sent to the
   output unchanged, but markups are processed, expanded to their results, and then written
   to the output file as strings (that is, with the str function, not repr). The act of
   processing EmPy source and handling markups is called "expansion."

   Code that is processed is executed exactly as if it were entered into the Python
   interpreter; that is, it is executed with the equivalent of eval (for expressions) and
   exec (for statements). For instance, inside an expression, abc represents the name abc,
   not the string "abc", just as it would in normal Python code.

   By default the embedding token prefix is the at sign (@), which appears neither in valid
   Python code nor commonly in English text; it can be overridden with the -p option (or with
   the empy.setPrefix function). The token prefix indicates to the EmPy interpreter that a
   special sequence follows and should be processed rather than sent to the output untouched
   (to indicate a literal at sign, it can be doubled as in @@).

   When the interpreter starts processing its target file, no modules are imported by
   default, save the empy pseudomodule (see below), which is placed in the globals; the empy
   pseuodmodule is associated with a particular interpreter; it is important that it not be
   removed from that interpreter's globals, nor that it be shared with other interpreters
   running concurrently. The globals are not cleared or reset in any way. It is perfectly
   legal to set variables or explicitly import modules and then use them in later markups,
   e.g., @{import time} ... @time.time(). Scoping rules are as in normal Python, although all
   defined variables and objects are taken to be in the global namespace.

   Activities you would like to be done before any processing of the main EmPy file can be
   specified with the -I, -D, -E, -F, and -P options. -I imports modules, -D executes a
   Python variable assignment, -E executes an arbitrary Python (not EmPy) statement, -F
   executes a Python (not EmPy) file, and -P processes an EmPy (not Python) file. These
   operations are done in the order they appear on the command line; any number of each
   (including, of course, zero) can be used.

Expansions

   The following markups are supported. For concreteness below, @ is taken for the sake of
   argument to be the prefix character, although this can be changed.

   @# COMMENT NEWLINE
          A comment. Comments, including the trailing newline, are stripped out completely.
          Comments should only be present outside of expansions. The comment itself is not
          processed in any way: It is completely discarded. This allows @# comments to be
          used to disable markups. Note: As special support for "bangpaths" in UNIX like
          operating systems, if the first line of a file (or indeed any context) begins with
          #!, and the interpreter has a processBangpaths option set to true (default), it is
          treated as a @# comment. A #! sequence appearing anywhere else will be handled
          literally and unaltered in the expansion. Example:

          @# This line is a comment.
          @# This will NOT be expanded: @x.

   @ WHITESPACE
          A @ followed by one whitespace character (a space, horizontal tab, vertical tab,
          carriage return, or newline) is expanded to nothing; it serves as a way to
          explicitly separate two elements which might otherwise be interpreted as being the
          same symbol (such as @name@ s to mean '@(name)s'; see below). Also, since a newline
          qualifies as whitespace here, the lone @ at the end of a line represents a line
          continuation, similar to the backslash in other languages. Coupled with statement
          expansion below, spurious newlines can be eliminated in statement expansions by use
          of the @{...}@ construct. Example:

          This will appear as one word: salt@ water.
          This is a line continuation; @
          this text will appear on the same line.

   @\ ESCAPE_CODE
          An escape code. Escape codes in EmPy are similar to C-style escape codes, although
          they all begin with the prefix character. Valid escape codes include:

        @\0
                NUL, null

        @\a
                BEL, bell

        @\b
                BS, backspace

        @\d
                three-digital decimal code DDD

        @\e
                ESC, escape

        @\f
                FF, form feed

        @\h
                DEL, delete

        @\n
                LF, linefeed character, newline

        @\oOOO
                three-digit octal code OOO

        @\qQQQQ
                four-digit quaternary code QQQQ

        @\r
                CR, carriage return

        @\s
                SP, space

        @\t
                HT, horizontal tab

        @\v
                VT, vertical tab

        @\xHH
                two-digit hexadecimal code HH

        @\z
                EOT, end of transmission

        @^X
                the control character ^X

          Unlike in C-style escape codes, escape codes taking some number of digits afterward
          always take the same number to prevent ambiguities. Furthermore, unknown escape
          codes are treated as parse errors to discourage potential subtle mistakes. Unlike
          in C, to represent an octal value, one must use @\o.... Example:

          This embeds a newline.@\nThis is on the following line.
          This beeps!@\a
          There is a tab here:@\tSee?
          This is the character with octal code 141: @\o141.

   @@
          A literal at sign (@). To embed two adjacent at signs, use @@@@, and so on. Any
          literal at sign that you wish to appear in your text must be written this way, so
          that it will not be processed by the system. Note: If a prefix other than @ has
          been chosen via the command line option, one expresses that literal prefix by
          doubling it, not by appending a @. Example:

          The prefix character is @@.
          To get the expansion of x you would write @@x.

   @), @], @}
          These expand to literal close parentheses, close brackets, and close braces,
          respectively; these are included for completeness and explicitness only. Example:

          This is a close parenthesis: @).

   @( EXPRESSION )
          Evaluate an expression, and replace the tokens with the string (via a call to str)
          representation evaluation of that expression. Whitespace immediately inside the
          parentheses is ignored; @( expression ) is equivalent to @(expression). If the
          expression evaluates to None, nothing is expanded in its place; this allows
          function calls that depend on side effects (such as printing) to be called as
          expressions. (If you really do want a None to appear in the output, then use the
          Python string "None".) Example:

          2 + 2 is @(2 + 2).
          4 squared is @(4**2).
          The value of the variable x is @(x).
          This will be blank: @(None).

   @( TEST ? THEN (: ELSE)_opt ($ CATCH)_opt )
          A special form of expression evaluation representing conditional and protected
          evaluation. Evaluate the "test" expression; if it evaluates to true (in the
          Pythonic sense), then evaluate the "then" section as an expression and expand with
          the str of that result. If false, then the "else" section is evaluated and
          similarly expanded. The "else" section is optional and, if omitted, is equivalent
          to None (that is, no expansion will take place).

          If the "catch" section is present, then if any of the prior expressions raises an
          exception when evaluated, the expansion will be substituted with the evaluation of
          the catch expression. (If the "catch" expression itself raises, then that exception
          will be propagated normally.) The catch section is optional and, if omitted, is
          equivalent to None (that is, no expansion will take place). An exception (cough) to
          this is if one of these first expressions raises a SyntaxError; in that case the
          protected evaluation lets the error through without evaluating the "catch"
          expression. The intent of this construct is to catch runtime errors, and if there
          is actually a syntax error in the "try" code, that is a problem that should
          probably be diagnosed rather than hidden. Example:

          What is x? x is @(x ? "true" : "false").
          Pluralization: How many words? @x word@(x != 1 ? 's').
          The value of foo is @(foo $ "undefined").
          The square root of -1 is @(math.sqrt(-1) $ "not real").

   @ SIMPLE_EXPRESSION
          As a shortcut for the @(...) notation, the parentheses can be omitted if it is
          followed by a "simple expression." A simple expression consists of a name followed
          by a series of function applications, array subscriptions, or attribute
          resolutions, with no intervening whitespace. For example:

          + a name, possibly with qualifying attributes (e.g., @value, @os.environ).
          + a straightforward function call (e.g., @min(2, 3), @time.ctime()), with no space
            between the function name and the open parenthesis.
          + an array subscription (e.g., '@array[8][index]', '@os.environ[9][name]', with no
            space between the name and the open bracket.
          + any combination of the above (e.g., '@function(args).attr[10][sub].other[11][i]
            (foo)').

          In essence, simple expressions are expressions that can be written ambiguously from
          text, without intervening space. Note that trailing dots are not considered part of
          the expansion (e.g., @x. is equivalent to @(x)., not @(x.), which would be illegal
          anyway). Also, whitespace is allowed within parentheses or brackets since it is
          unambiguous , but not between identifiers and parentheses, brackets, or dots.
          Explicit @(...) notation can be used instead of the abbreviation when concatenation
          is what one really wants (e.g., @(word)s for simple pluralization of the contents
          of the variable word). As above, if the expression evaluates to the None object,
          nothing is expanded. Example:

          The value of x is @x.
          The ith value of a is @a[i].
          The result of calling f with q is @f(q).
          The attribute a of x is @x.a.
          The current time is @time.ctime(time.time()).
          The current year is @time.localtime(time.time())[0].
          These are the same: @min(2,3) and @min(2, 3).
          But these are not the same: @min(2, 3) vs. @min (2, 3).
          The plural of @name is @(name)s, or @name@ s.

   @` EXPRESSION `
          Evaluate a expression, and replace the tokens with the repr (instead of the str
          which is the default) of the evaluation of that expression. This expansion is
          primarily intended for debugging and is unlikely to be useful in actual practice.
          That is, a @`...` is identical to @(repr(...)). Example:

          The repr of the value of x is @`x`.
          This print the Python repr of a module: @`time`.
          This actually does print None: @`None`.

   @: EXPRESSION : DUMMY :
          Evaluate an expression and then expand to a @:, the original expression, a :, the
          evaluation of the expression, and then a :. The current contents of the dummy area
          are ignored in the new expansion. In this sense it is self-evaluating; the syntax
          is available for use in situations where the same text will be sent through the
          EmPy processor multiple times. Example:

          This construct allows self-evaluation:
          @:2 + 2:this will get replaced with 4:

   @[ noop : IGNORED ]
          The material contained within the substitution is completely ignored. The
          substiution does not expand to anything, and indeed expansion contained within the
          ignored block are not expanded. This is included simply for completeness, and can
          served as a block comment. Example:

          @[noop:
          All this stuff would appear here
          if it weren't for the noop.
          @{
          while 1:
              print "Testing"
          }@
          ]

   @[ if EXPRESSION : CODE ]
          Evaluate the Python test expression; if it evaluates to true, then expand the
          following code through the EmPy system (which can contain markups), otherwise,
          expand to nothing. Example:

          @[if x > 0:@x is positive.]
          @# If you want to embed unbalanced right brackets:
          @[if showPrompt:@\x5dINIT HELLO]

   @[ while EXPRESSION : CODE ]
          Evaluate the Python expression; if it evaluates to true, then expand the code and
          repeat; otherwise stop expanding. Example:

          @[while i < 10:@ i is @i.@\n]

   @[ for NAME in EXPRESSION : CODE ]
          Evaluate the Python expression and treat it as a sequence; iterate over the
          sequence, assigning each element to the provided name in the globals, and expanding
          the given code each time. Example:

          @[for i in range(5):@ The cube of @i is @(i**3).@\n]

   @[ macro SIGNATURE : CODE ]
          Define a "macro," which is a function-like object that causes an expansion whenever
          it is called. The signature defines the name of the function and its parameter
          list, if any -- just like normal Python functions, macro signatures can include
          optional arguments, keyword arguments, etc. When defined, calling the macro results
          in the given code to be expanded, with the function arguments involved as the
          locals dictionary in the expansion. Additionally, the doc string of the function
          object that is created corresponds to the expansion. Example:

          @[macro f(n):@ @[for i in range(n):@ @i**2 is @(i**2)@\n]]

   @{ STATEMENTS }
          Execute a (potentially compound) statement; statements have no return value, so the
          expansion is not replaced with anything. Multiple statements can either be
          separated on different lines, or with semicolons; indentation is significant, just
          as in normal Python code. Statements, however, can have side effects, including
          printing; output to sys.stdout (explicitly or via a print statement) is collected
          by the interpreter and sent to the output. The usual Python indentation rules must
          be followed, although if the statement consists of only one statement, leading and
          trailing whitespace is ignored (e.g., @{ print time.time() } is equivalent to
          @{print time.time()}). Example:

          @{x = 123}
          @{a = 1; b = 2}
          @{print time.time()}
          @# Note that extra newlines will appear above because of the
          @# newlines trailing the close braces.  To suppress them
          @# use a @ before the newline:
          @{
          for i in range(10):
              print "i is %d" % i
          }@
          @{print "Welcome to EmPy."}@

   @% KEY (WHITESPACE VALUE)_opt NEWLINE
          Declare a significator. Significators consume the whole line (including the
          trailing newline), and consist of a key string containing no whitespace, and than
          optional value prefixed by whitespace. The key may not start with or contain
          internal whitespace, but the value may; preceding or following whitespace in the
          value is stripped. Significators are totally optional, and are intended to be used
          for easy external (that is, outside of EmPy) identification when used in large
          scale environments with many EmPy files to be processed. The purpose of
          significators is to provide identification information about each file in a
          special, easy-to-parse form so that external programs can process the significators
          and build databases, independently of EmPy. Inside of EmPy, when a significator is
          encountered, its key, value pair is translated into a simple assignment of the form
          __KEY__ = VALUE , where "__KEY__" is the key string with two underscores on either
          side and "VALUE" is a Python expression. Example:

          @%title     "Nobody knows the trouble I've seen"
          @%keywords  ['nobody', 'knows', 'trouble', 'seen']
          @%copyright [2000, 2001, 2002]

Substitutions

   Supported are conditional and repeated substitutions, which involve testing or iterating
   over Python expressions and then possibly expanding EmPy code. These different from normal
   Python if, for, and while statements since the result is an EmPy expansion, rather than
   the execution of a Python statement; the EmPy expansion may, of course, contain further
   expansions. This is useful for in-place conditional or repeated expansion of similar text;
   as with all expansions, markups contained within the EmPy code are processed. The simplest
   form would consist something like:
        @[if x != 0:x is @x]

   This will expand x is @x if x is greater than zero. Note that all characters, including
   whitespace and newlines, after the colon and before the close bracket are considered part
   of the code to be expanded; to put a space in there for readability, you can use the
   prefix and a whitespace character:
        @[if x != 0:@ x is @x]

   Iteration via while is also possible:
        @{i = 0}@[while i < 10:@ i is @i@\n@{i = i + 1}]

   This is a rather contrived example which iterates i from 0 to 9 and then prints "i is
   (value)" for each iteration.

   A more practical example can be demonstrated with the for notation:
        <table>@[for x in elements:@ <tr><td>@x</td></tr>]</table>

   This EmPy fragment would format the contents of elements into an HTML table, with one
   element per row.

   The macro substitution doesn't get replaced with anything, but instead defines a "macro,"
   or recallable expansion, which looks and behaves like a function. When called, it expands
   its contents. The arguments to the function -- which can be defined with optional,
   remaining, and keyword arguments, just like any Python function -- can be referenced in
   the expansion as local variables. For concreteness, the doc string of the macro function
   is the original expansion. An macro substitution of the form @[macro SIGNATURE:CODE] is
   equivalent to the following Python code:
        def SIGNATURE:
            repr(CODE) # so it is a doc string
            empy.string(repr(CODE), '<macro>', locals())

   This can be used to defer the expansion of something to a later time:
        @[macro header(title='None'):<head><title>@title</title></head>]

   Note that all text up to the trailing bracket is considered part of the EmPy code to be
   expanded. If one wishes a stray trailing brackets to appear in the code, one can use an
   escape code to indicate it, such as @\x5d. Matching open and close bracket pairs do not
   need to be escaped, for either bracket pairs in an expansion or even for further
   substitutions:
        @[if something:@ This is an unbalanced close bracket: @\x5d]
        @[if something:@ This is a balanced bracket pair: [word]]
        @[if something:@ @[if somethingElse:@ This is nested.]]

Significators

   Significators are intended to represent special assignment in a form that is easy to
   externally parse. For instance, if one has a system that contains many EmPy files, each of
   which has its own title, one could use a title significator in each file and use a simple
   regular expression to find this significator in each file and organize a database of the
   EmPy files to be built. This is an easier proposition than, for instance, attempting to
   grep for a normal Python assignment (inside a @{...} expansion) of the desired variable.

   Significators look like the following:
        @%KEY VALUE

   including the trailing newline, where "key" is a name and "value" is a Python expression,
   and are separated by any whitespace. This is equivalent to the following Python code:
        __KEY__ = VALUE

   That is to say, a significator key translates to a Python variable consisting of that key
   surrounded by double underscores on either side. The value may contain spaces, but the key
   may not. So:
        @%title "All Roads Lead to Rome"

   translates to the Python code:
        __title__ = "All Roads Lead to Rome"

   but obviously in a way that easier to detect externally than if this Python code were to
   appear somewhere in an expansion. Since significator keys are surrounded by double
   underscores, significator keys can be any sequence of alphanumeric and underscore
   characters; choosing 123 is perfectly valid for a significator (although straight), since
   it maps to the name __123__ which is a legal Python identifier.

   Note the value can be any Python expression. The value can be omitted; if missing, it is
   treated as None.

   Significators are completely optional; it is totally legal for a EmPy file or files to be
   processed without containing any significators.

   A regular expression string designed to match significators (with the default prefix) is
   available as empy.SIGNIFICATOR_RE_STRING, and also is a toplevel definition in the em
   module itself.

Diversions

   EmPy supports an extended form of m4-style diversions, which are a mechanism for deferring
   and recalling output on demand. Multiple "streams" of output can be diverted and
   undiverted in this manner. A diversion is identified with a name, which is any immutable
   object such an integer or string. When recalled, diverted code is not resent through the
   EmPy interpreter (although a filter could be set up to do this).

   By default, no diversions take place. When no diversion is in effect, processing output
   goes directly to the specified output file. This state can be explicitly requested at any
   time by calling the empy.stopDiverting function. It is always legal to call this function.

   When diverted, however, output goes to a deferred location which can then be recalled
   later. Output is diverted with the empy.startDiversion function, which takes an argument
   that is the name of the diversion. If there is no diversion by that name, a new diversion
   is created and output will be sent to that diversion; if the diversion already exists,
   output will be appended to that preexisting diversion.

   Output send to diversions can be recalled in two ways. The first is through the
   empy.playDiversion function, which takes the name of the diversion as an argument. This
   recalls the named diversion, sends it to the output, and then erases that diversion. A
   variant of this behavior is the empy.replayDiversion, which recalls the named diversion
   but does not eliminate it afterwards; empy.replayDiversion can be repeatedly called with
   the same diversion name, and will replay that diversion repeatedly. empy.createDiversion
   create a diversion without actually diverting to it, for cases where you want to make sure
   a diversion exists but do not yet want to send anything to it.

   The diversion object itself can be retrieved with empy.retrieveDiversion. Diversions act
   as writable file-objects, supporting the usual write, writelines, flush, and close
   methods. The data that has been diverted to them can be retrieved in one of two ways;
   either through the asString method, which returns the entire contents of the diversion as
   a single strong, or through the asFile method, which returns the contents of the diversion
   as a readable (not writable) file-like object.

   Diversions can also be explicitly deleted without recalling them with the
   empy.purgeDiversion function, which takes the desired diversion name as an argument.

   Additionally there are three functions which will apply the above operations to all
   existing diversions: empy.playAllDiversions, empy.replayAllDiversions, and
   empy.purgeAllDiversions. All three will do the equivalent of a empy.stopDiverting call
   before they do their thing.

   The name of the current diversion can be requested with the empy.getCurrentDiversion
   function; also, the names of all existing diversions (in sorted order) can be retrieved
   with empy.getAllDiversions.

   When all processing is finished, the equivalent of a call to empy.playAllDiversions is
   done.

Filters

   EmPy also supports dynamic filters. Filters are put in place right "before" the final
   output file, and so are only invoked after all other processing has taken place (including
   interpreting and diverting). Filters take input, remap it, and then send it to the output.

   The current filter can be retrieved with the empy.getFilter function. The filter can be
   cleared (reset to no filter) with empy.resetFilter and a special "null filter" which does
   not send any output at all can be installed with empy.nullFilter. A custom filter can be
   set with the empy.setFilter function; for convenience, specialized forms of filters
   preexist and can be accessed with shortcuts for the empy.setFilter argument:
     * None is a special filter meaning "no filter"; when installed, no filtering whatsoever
       will take place. empy.setFilter(None) is equivalent to empy.resetFilter().
     * 0 (or any other numeric constant equal to zero) is another special filter that
       represents the null filter; when installed, no output will ever be sent to the
       filter's sink.
     * A filter specified as a function (or lambda) is expected to take one string argument
       and return one string argument; this filter will execute the function on any input and
       use the return value as output.
     * A filter that is a string is a 256-character table is substituted with the result of a
       call to string.translate using that table.
     * A filter can be an instance of a subclass of empy.Filter. This is the most general
       form of filter. (In actuality, it can be any object that exhibits a Filter interface,
       which would include the normal file-like write, flush, and close methods, as well as
       next, attach, and detach methods for filter-specific behavior.)
     * Finally, the argument to empy.setFilter can be a Python list consisting of one or more
       of the above objects. In that case, those filters are chained together in the order
       they appear in the list. An empty list is the equivalent of 'None'; all filters will
       be uninstalled.

   Filters are, at their core, simply file-like objects (minimally supporting write, flush,
   and close methods that behave in the usual way) which, after performing whatever
   processing they need to do, send their work to the next file-like object or filter in
   line, called that filter's "sink." That is to say, filters can be "chained" together; the
   action of each filter takes place in sequence, with the output of one filter being the
   input of the next. Additionally, filters support a _flush method (note the leading
   underscore) which will always flush the filter's underlying sink; this method should be
   not overridden.

   Filters also support three additional methods, not part of the traditional file interface:
   attach, which takes as an argument a file-like object (perhaps another filter) and sets
   that as the filter's "sink" -- that is, the next filter/file-like object in line. detach
   (which takes no arguments) is another method which flushes the filter and removes its
   sink, leaving it isolated. Finally, next is an accessor method which returns the filter's
   sink -- or None, if the filter does not yet have a sink attached.

   To create your own filter, you can create an object which supports the above described
   interface, or simply derive from the empy.Filter class and override its write and possibly
   flush methods. You can chain filters together by passing them as elements in a list to the
   empy.setFilter function, or you can chain them together manually with the attach method:
        firstFilter.attach(secondFilter)
        empy.setFilter(firstFilter)

   or just let EmPy do the chaining for you:
        empy.setFilter([firstFilter, secondFilter])

   In either case, EmPy will walk the filter chain and find the end and then hook that into
   the appropriate interpreter stream; you need not do this manually.

   Subclasses of empy.Filter are already provided with the above null, function, and string
   functionality described above; they are NullFilter, FunctionFilter, and StringFilter,
   respectively. In addition, a filter which supports buffering, BufferedFilter, is provided.
   Several variants are included: SizeBufferedFilter, a filter which buffers into fixed-sized
   chunks, LineBufferedFilter, a filter which buffers by lines, and MaximallyBufferedFilter,
   a filter which completely buffers its input.

Hooks

   The EmPy system also allows for the usage of "hooks," which are callbacks that can be
   registered with an interpreter to get information on the current state of activity and act
   upon it.

   Hooks are associated with names, which are merely strings; these strings represent a state
   of the interpreter. Any number of hooks can be associated with a given name, and are
   registered with the empy.addHook function call. Hooks are callable objects which take two
   arguments: first, a reference to the interpreter that is running; and second, a dictionary
   that contains contextual information about the point at which the hook is invoked; the
   contents of this dictionary are dependent on the hook name.

   Hooks can perform any reasonable action, with one caveat: When hooks are invoked,
   sys.stdout may not be properly wrapped and so should be considered unusable. If one wishes
   to really write to the actually stdout stream (not the interpreter), use
   sys.__stdout__.write. If one wishes to send output to the interpreter, then use
   interpreter.write. Neither references to sys.stdout nor print statements should ever
   appear in a hook.

   The hooks associated with a given name can be retrieved with empy.getHooks. All hooks
   associated with a name can be cleared with empy.clearHooks, and all hooks associated with
   all names can be cleared with empy.clearAllHooks. A hook added with empy.addHook can be
   removed with empy.removeHook. Finally, hooks can be manually invoked via empy.invokeHook.

   The following hooks are supported; also listed in curly braces are the keys contained in
   the dictionary argument:

   at_shutdown
          The interpreter is shutting down.

   at_handle {meta}
          An exception is being handled; meta is the exception (an instance of MetaError).
          Note that this hook is invoked when the exception is handled by the EmPy system,
          not when it is thrown.

   before_include {name, file}
          An empy.include call is about to be processed; name is the context name of the
          inclusion and file is the actual file object associated with the include.

   after_include
          An empy.include was just completed.

   before_expand {string, locals}
          An empy.expand call is about to be processed. string is the actual data that is
          about to be processed; locals is the locals dictionary or None.

   after_expand
          An empy.expand was just completed.

   at_quote {string}
          An empy.quote call is about to be processed; string is the string to be quoted.

   at_escape {string}
          An empy.escape call is about to be processed; string is the string to be escaped.

   before_file {name, file}
          A file object is just about to be processed. name is the context name associated
          with the object and file is the file object itself.

   after_file
          A file object has just finished processing.

   before_string {name, string}
          A standalone string is just about to be processed. name is the context name
          associated with it and string is the string itself.

   after_string
          A standalone string has just finished being processed.

   at_parse {scanner}
          A parsing pass is just about to be performed. scanner is the scanner associated
          with the parsing pass.

   before_evaluate {expression, locals}
          A Python expression is just about to be evaluated. expression is the (string)
          expression, and locals is the locals dictionary or None.

   after_evaluate
          A Python expression was just evaluated.

   before_execute {statements, locals}
          A chunk of Python statements is just about to be evaluated. statements is the
          (string) statement block, and locals is the locals dictionary or None.

   before_single {source, locals}
          A single interactive source code fragment (just as in the Python interpreter) is
          about to be executed via Interpreter.single. source is the code (expression or
          statement) to execute, and locals is the locals directory or None.

   after_single
          A single has just taken place.

   before_substitute {substitution}
          A @[...] substitution is just about to be done. substitution is the substitution
          string itself.

   after_substitute
          A substitution just took place.

   before_significate {key, value}
          A significator is just about to be processed; key is the key and value is the
          value.

   after_significate
          A significator was just processed.

   As a practical example, this sample Python code would print a pound sign followed by the
   name of every file that is included with 'empy.include':
        def includeHook(interpreter, keywords):
            interpreter.write("# %s\n" % keywords['name'])
        empy.addHook('before_include', includeHook)

   Note that this snippet properly uses a call to interpreter.write instead of executing a
   print statement.

Data flow

   input -> interpreter -> diversions -> filters -> output

   Here, in summary, is how data flows through a working EmPy system:

    1. Input comes from a source, such an .em file on the command line, or via an
       empy.include statement.
    2. The interpreter processes this material as it comes in, expanding token sequences as
       it goes.
    3. After interpretation, data is then sent through the diversion layer, which may allow
       it directly through (if no diversion is in progress) or defer it temporarily.
       Diversions that are recalled initiate from this point.
    4. Any filters in place are then used to filter the data and produce filtered data as
       output.
    5. Finally, any material surviving this far is sent to the output stream. That stream is
       stdout by default, but can be changed with the -o or -a options, or may be fully
       buffered with the -B option (that is, the output file would not even be opened until
       the entire system is finished).

  Pseudomodule contents

   The empy pseudomodule (available only in an operating EmPy system) contains the following
   functions and objects (and their signatures, with a suffixed opt indicating an optional
   argument):

   First, basic identification:

   VERSION
          A constant variable which contains a string representation of the EmPy version.

   SIGNIFICATOR_RE_STRING
          A constant variable representing a regular expression string that can be used to
          find significators in EmPy code.

   interpreter
          The instance of the interpreter that is currently being used to perform execution.

   argv
          A list consisting of the name of the primary EmPy script and its command line
          arguments, in analogue to the sys.argv list.

   args
          A list of the command line arguments following the primary EmPy script; this is
          equivalent to empy.argv[1:].

   identify() -> string, integer
          Retrieve identification information about the current parsing context. Returns a
          2-tuple consisting of a filename and a line number; if the file is something other
          than from a physical file (e.g., an explicit expansion with empy.expand, a
          file-like object within Python, or via the -E or -F command line options), a string
          representation is presented surrounded by angle brackets. Note that the context
          only applies to the EmPy context, not the Python context.

   setName(name)
          Manually set the name of the current context.

   setLine(line)
          Manually set the line number of the current context; line must be a numeric value.
          Note that afterward the line number will increment by one for each newline that is
          encountered, as before.

   atExit(callable)
          Register a callable object (or function) taking no arguments which will be called
          at the end of a normal shutdown. Callable objects registered in this way are called
          in the reverse order in which they are added, so the first callable registered with
          empy.atExit is the last one to be called. Note that although the functionality is
          related to hooks, empy.atExit does no work via the hook mechanism, and you are
          guaranteed that the interpreter and stdout will be in a consistent state when the
          callable is invoked.

   Globals manipulation:

   getGlobals()
          Retrieve the globals dictionary for this interpreter. Unlike when calling globals()
          in Python, this dictionary can be manipulated and you can expect changes you make
          to it to be reflected in the interpreter that holds it.

   setGlobals(globals)
          Reseat the globals dictionary associated with this interpreter to the provided
          mapping type.

   updateGlobals(globals)
          Merge the given dictionary into this interpreter's globals.

   clearGlobals(globals_opt)
          Clear out the globals (restoring, of course, the empy pseudomodule). Optionally,
          instead of starting with a refresh dictionary, use the dictionary provided.

   Filter classes:

   Filter
          The base Filter class which can be derived from to make custom filters.

   NullFilter
          A null filter; all data sent to the filter is discarded.

   FunctionFilter
          A filter which uses a function taking a string and returning another to perform the
          filtering.

   StringFilter
          A filter which uses a 256-character string table to map any incoming character.

   BufferedFilter
          A filter which does not modify its input, but instead holds it until it is told to
          flush (via the filter's flush method). This also serves as the base class for the
          other buffered filters below.

   SizeBufferedFilter
          A filter which buffers into fixed-size chunks, with the possible exception of the
          last chunk. The buffer size is indicated as the sole argument to the constructor.

   LineBufferedFilter
          A filter which buffers into lines, with the possible exception of the last line
          (which may not end with a newline).

   MaximallyBufferedFilter
          A filter which does not flush any of its contents until it is closed. Note that
          since this filter ignores calls to its flush method, this means that installing
          this filter and then replacing it with another can result in loss of data.

   The following functions allow direct execution; optional locals arguments, if specified,
   are treated as the locals dictionary in evaluation and execution:

   evaluate(expression, locals_opt)
          Evaluate the given expression.

   execute(statements, locals_opt)
          Execute the given statement(s).

   single(source, locals_opt)
          Interpret the "single" source code, just as the Python interactive interpreter
          would.

   substitute(substitution, locals_opt)
          Perform the given substitution.

   significate(key, value_opt)
          Do a manual signification. If value is not specified, it is treated as None.

   The following functions relate to source manipulation:

   include(file_or_filename, locals_opt)
          Include another EmPy file, by processing it in place. The argument can either be a
          filename (which is then opened with open in text mode) or a file object, which is
          used as is. Once the included file is processed, processing of the current file
          continues. Includes can be nested. The call also takes an optional locals
          dictionary which will be passed into the evaluation function.

   expand(string, locals_opt) -> string
          Explicitly invoke the EmPy parsing system to process the given string and return
          its expansion. This allows multiple levels of expansion, e.g., @(empy.expand("@(2 +
          2)")). The call also takes an optional locals dictionary which will be passed into
          the evaluation function. This is necessary when text is being expanded inside a
          function definition and it is desired that the function arguments (or just plain
          local variables) are available to be referenced within the expansion.

   quote(string) -> string
          The inverse process of empy.expand, this will take a string and return a new string
          that, when expanded, would expand to the original string. In practice, this means
          that appearances of the prefix character are doubled, except when they appear
          inside a string literal.

   escape(string, more_opt) -> string
          Given a string, quote the nonprintable characters contained within it with EmPy
          escapes. The optional more argument specifies additional characters that should be
          escaped.

   flush()
          Do an explicit flush on the underlying stream.

   string(string, name_opt, locals_opt)
          Explicitly process a string-like object. This differs from empy.expand in that the
          string is directly processed into the EmPy system, rather than being evaluated in
          an isolated context and then returned as a string.

   Changing the behavior of the pseudomodule itself:

   flatten(keys_opt)
          Perform the equivalent of from empy import ... in code (which is not directly
          possible because empy is a pseudomodule). If keys is omitted, it is taken as being
          everything in the empy pseudomodule. Each of the elements of this pseudomodule is
          flattened into the globals namespace; after a call to empy.flatten, they can be
          referred to simple as globals, e.g., @divert(3) instead of @empy.divert(3). If any
          preexisting variables are bound to these names, they are silently overridden. Doing
          this is tantamount to declaring an from ... import ... which is often considered
          bad form in Python.

   Prefix-related functions:

   getPrefix() -> char
          Return the current prefix.

   setPrefix(char)
          Set a new prefix. Immediately after this call finishes, the prefix will be changed.
          Changing the prefix affects only the current interpreter; any other created
          interpreters are unaffected.

   Diversions:

   stopDiverting()
          Any diversions that are currently taking place are stopped; thereafter, output will
          go directly to the output file as normal. It is never illegal to call this
          function.

   createDiversion(name)
          Create a diversion, but do not begin diverting to it. This is the equivalent of
          starting a diversion and then immediately stopping diversion; it is used in cases
          where you want to make sure that a diversion will exist for future replaying but
          may be empty.

   startDiversion(name)
          Start diverting to the specified diversion name. If such a diversion does not
          already exist, it is created; if it does, then additional material will be appended
          to the preexisting diversions.

   playDiversion(name)
          Recall the specified diversion and then purge it. The provided diversion name must
          exist.

   replayDiversion(name)
          Recall the specified diversion without purging it. The provided diversion name must
          exist.

   purgeDiversion(name)
          Purge the specified diversion without recalling it. The provided diversion name
          must exist.

   playAllDiversions()
          Play (and purge) all existing diversions in the sorted order of their names. This
          call does an implicit empy.stopDiverting before executing.

   replayAllDiversions()
          Replay (without purging) all existing diversions in the sorted order of their
          names. This call does an implicit empy.stopDiverting before executing.

   purgeAllDiversions()
          Purge all existing diversions without recalling them. This call does an implicit
          empy.stopDiverting before executing.

   getCurrentDiversion() -> diversion
          Return the name of the current diversion.

   getAllDiversions() -> sequence
          Return a sorted list of all existing diversions.

   Filters:

   getFilter() -> filter
          Retrieve the current filter. None indicates no filter is installed.

   resetFilter()
          Reset the filter so that no filtering is done.

   nullFilter()
          Install a special null filter, one which consumes all text and never sends any text
          to the output.

   setFilter(filter)
          Install a new filter. A filter is None or an empty sequence representing no filter,
          or 0 for a null filter, a function for a function filter, a string for a string
          filter, or an instance of empy.Filter. If filter is a list of the above things,
          they will be chained together manually; if it is only one, it will be presumed to
          be solitary or to have already been manually chained together. See the "Filters"
          section for more information.

   Hooks:

   enableHooks()
          Enable invocation of hooks. By default hooks are enabled.

   disableHooks()
          Disable invocation of hooks. Hooks can still be added, removed, and queried, but
          invocation of hooks will not occur (even explicit invocation with empy.invokeHook).

   areHooksEnabled()
          Return whether or not hooks are presently enabled.

   getHooks(name)
          Get a list of the hooks associated with this name.

   clearHooks(name)
          Clear all hooks associated with this name.

   clearAllHooks(name)
          Clear all hooks associated with this name.

   addHook(name, hook, prepend_opt)
          Add this hook to the hooks associated with this name. By default, the hook is
          appended to the end of the existing hooks, if any; if the optional insert argument
          is present and true, it will be prepended to the list instead.

   removeHook(name, hook)
          Remove this hook from the hooks associated with this name.

   invokeHook(name_, ...)
          Manually invoke all the hooks associated with this name. The remaining arguments
          are treated as keyword arguments and the resulting dictionary is passed in as the
          second argument to the hooks.

  Invocation

   Basic invocation involves running the interpreter on an EmPy file and some optional
   arguments. If no file are specified, or the file is named -, EmPy takes its input from
   stdin. One can suppress option evaluation (to, say, specify a file that begins with a
   dash) by using the canonical -- option.

   -a/--append (filename)
          Open the specified file for append instead of using stdout.

   -f/--flatten
          Before processing, move the contents of the empy pseudomodule into the globals,
          just as if empy.flatten() were executed immediately after starting the interpreter.
          That is, e.g., empy.include can be referred to simply as include when this flag is
          specified on the command line. This can also be specified through the existence of
          the EMPY_FLATTEN environment variable.

   -h/--help
          Print usage and exit.

   -H/--extended-help
          Print extended usage and exit. Extended usage includes a rundown of all the legal
          expansions, escape sequences, pseudomodule contents, used hooks, and supported
          environment variables.

   -i/--interactive
          After the main EmPy file has been processed, the state of the interpreter is left
          intact and further processing is done from stdin. This is analogous to the Python
          interpreter's -i option, which allows interactive inspection of the state of the
          system after a main module is executed. This behaves as expected when the main file
          is stdin itself. This can also be specified through the existence of the
          EMPY_INTERACTIVE environment variable.

   -k/--suppress-errors
          Normally when an error is encountered, information about its location is printed
          and the EmPy interpreter exits. With this option, when an error is encountered
          (except for keyboard interrupts), processing stops and the interpreter enters
          interactive mode, so the state of affairs can be assessed. This is also helpful,
          for instance, when experimenting with EmPy in an interactive manner. -k implies -i.

   -o/--output (filename)
          Open the specified file for output instead of using stdout. If a file with that
          name already exists it is overwritten.

   -p/--prefix (prefix)
          Change the prefix used to detect expansions. The argument is the one-character
          string that will be used as the prefix. Note that whatever it is changed to, the
          way to represent the prefix literally is to double it, so if $ is the prefix, a
          literal dollar sign is represented with $$. Note that if the prefix is changed to
          one of the secondary characters (those that immediately follow the prefix to
          indicate the type of action EmPy should take), it will not be possible to represent
          literal prefix characters by doubling them (e.g., if the prefix were unadvisedly
          changed to # then ## would already have to represent a comment, so ## could not
          represent a literal #). This can also be specified through the EMPY_PREFIX
          environment variable.

   -r/--raw-errors
          Normally, EmPy catches Python exceptions and prints them alongside an error
          notation indicating the EmPy context in which it occurred. This option causes EmPy
          to display the full Python traceback; this is sometimes helpful for debugging. This
          can also be specified through the existence of the EMPY_RAW_ERRORS environment
          variable.

   -B/--buffered-output
          Fully buffer processing output, including the file open itself. This is helpful
          when, should an error occur, you wish that no output file be generated at all (for
          instance, when using EmPy in conjunction with make). When specified, either the -o
          or -a options must be specified (and the -B option must precede them; full
          buffering does not work with stdout. This can also be specified through the
          existence of the EMPY_BUFFERED_OUTPUT environment variable.

   -D/--define (assignment)
          Execute a Python assignment of the form variable = expression. If only a variable
          name is provided (i.e., the statement does not contain an = sign), then it is taken
          as being assigned to None. The -D option is simply a specialized -E option that
          special cases the lack of an assignment operator. Multiple -D options can be
          specified.

   -E/--execute (statement)
          Execute the Python (not EmPy) statement before processing any files. Multiple -E
          options can be specified.

   -F/--execute-file (filename)
          Execute the Python (not EmPy) file before processing any files. This is equivalent
          to -E execfile("filename") but provides a more readable context. Multiple -F
          options can be specified.

   -I/--import (module)
          Imports the specified module name before processing any files. Multiple modules can
          be specified by separating them by commas, or by specifying multiple -I options.

   -P/--preprocess (filename)
          Process the EmPy file before processing the primary EmPy file on the command line.

   -V/--version
          Print version and exit.

  Environment variables

   EmPy also supports a few environment variables to predefine certain behaviors. The
   settings chosen by environment variables can be overridden via command line arguments. The
   following environment variables have meaning to EmPy:

   EMPY_OPTIONS
          If present, the contents of this environment variable will be treated as options,
          just as if they were entered on the command line, before the actual command line
          arguments are processed. Note that these arguments are not processed by the shell,
          so quoting, filename globbing, and the like, will not work.

   EMPY_PREFIX
          If present, the value of this environment variable represents the prefix that will
          be used; this is equivalent to the -p command line option.

   EMPY_FLATTEN
          If defined, this is equivalent to including -f on the command line.

   EMPY_RAW_ERRORS
          If defined, this is equivalent to including -r on the command line.

   EMPY_INTERACTIVE
          If defined, this is equivalent to including -i on the command line.

   EMPY_BUFFERED_OUTPUT
          If defined, this is equivalent to including -B on the command line.

  Examples and testing EmPy

   See the sample EmPy file sample.em which is included with the distribution. Run EmPy on it
   by typing something like (presuming a UNIX-like operating system):
         ./em.py sample.em

   and compare the results and the sample source file side by side. The sample content is
   intended to be self-documenting.

   The file sample.bench is the benchmark output of the sample. Running the EmPy interpreter
   on the provided sample.em file should produce precisely the same results. You can run the
   provided test script to see if your EmPy environment is behaving as expected:
        ./test.sh

   By default this will test with the first Python interpreter available in the path; if you
   want to test with another interpreter, you can provide it as the first argument on the
   command line, e.g.:
        ./test.sh python2.1
        ./test.sh /usr/bin/python1.5
        ./test.sh jython

  Embedding EmPy

   Embedding EmPy into your application is quite simple. The relative complexity of the
   em.invoke function is due to handling every possible combination of options (via the
   command line and environment variables). An EmPy interpreter can be created with as code
   as simple as:
        import em
        interpreter = em.Interpreter()
        # The following prints the results to stdout:
        interpreter.string("@{x = 123}@x\n")
        # This expands to the same thing, but puts the results as a
        # string in the variable result:
        result = interpreter.expand("@{x = 123}@x\n")
        # Process an actual file (and output to stdout):
        interpreter.file('/path/to/some/file')

   When you are finished with your interpreter, it is important to call its shutdown method:
        interpreter.shutdown()

   This will ensure that the interpreter cleans up all its overhead, entries in the
   sys.stdout proxy, and so forth. It is usually advisable that this be used in a
   try...finally clause:
        interpreter = em.Interpreter(...)
        try:
            ...
        finally:
            interpreter.shutdown()

   The em.Interpreter constructor takes the following arguments; all are optional:

   output
          The output file which the interpreter will be sending all its processed data to.
          This need only be a file-like object; it need not be an actual file. If omitted,
          sys.__stdout__ is used.

   argv
          An argument list analogous to sys.argv, consisting of the script name and zero or
          more arguments. These are available to executing interpreters via empy.argv and
          empy.args. If omitted, a non-descript script name is used with no arguments.

   prefix
          The single character prefix. Defaults to @.

   options
          A dictionary of options that can override the default behavior of the interpreter.
          The names of the options are constant names ending in _OPT and their defaults are
          given in Interpreter.DEFAULT_OPTIONS.

   globals
          By default, interpreters begin with a pristine dictionary of globals (except, of
          course, for the empy pseudomodule). Specifying this argument will allow the globals
          to start with more.

   Many things can be done with EmPy interpreters; for the full developer documentation, see
   the generated documentation for the em module.

  Interpreter options

   The following options (passed in as part of the options dictionary to the Interpreter
   constructor) have the following meanings. The defaults are shown below and are also
   indicated in an Interpreter.DEFAULT_OPTIONS dictionary.

   BANGPATH_OPT
          Should a bangpath (#!) as the first line of an EmPy file be treated as if it were
          an EmPy comment? Note that #! sequences starting lines or appearing anywhere else
          in the file are untouched regardless of the value of this option. Default: true.

   BUFFERED_OPT
          Should an abort method be called upon failure? This relates to the fully-buffered
          option, where all output can be buffered including the file open; this option only
          relates to the interpreter's behavior after that proxy file object has been
          created. Default: false.

   RAW_OPT
          Should errors be displayed as raw Python errors (that is, the exception is allowed
          to propagate through to the toplevel so that the user gets a standard Python
          traceback)? Default: false.

   EXIT_OPT
          Upon an error, should execution continue (although the interpreter stacks will be
          purged)? Note that even in the event this is set, the interpreter will halt upon
          receiving a KeyboardInterrupt. Default: true.

   FLATTEN_OPT
          Upon initial startup, should the empy pseudomodule namespace be flattened, i.e.,
          should empy.flatten be called? Note this option only has an effect when the
          interpreter is first created; thereafter it is ignored. Default: false.

  Known issues and caveats

     * EmPy was primarily intended for static processing of documents, rather than dynamic
       use, and hence speed of processing was not a major consideration in its design.
     * EmPy is not threadsafe.
     * Expressions (@(...)) are intended primarily for their return value; statements
       (@{...}) are intended primarily for their side effects, including of course printing.
       If an expression is expanded that as a side effect prints something, then the printing
       side effects will appear in the output before the expansion of the expression value.
     * Due to Python's curious handling of the print keyword -- particularly the form with a
       trailing comma to suppress the final newline -- mixing statement expansions using
       prints inline with unexpanded text will often result in surprising behavior, such as
       extraneous (sometimes even deferred!) spaces. This is a Python "feature," and occurs
       in non-EmPy applications as well; for finer control over output formatting, use
       sys.stdout.write or empy.interpreter.write (these will do the same thing) directly.
     * To function properly, EmPy must override sys.stdout with a proxy file object, so that
       it can capture output of side effects and support diversions for each interpreter
       instance. It is important that code executed in an environment not rebind sys.stdout,
       although it is perfectly legal to invoke it explicitly (e.g., @sys.stdout.write("Hello
       world\n")). If one really needs to access the "true" stdout, then use sys.__stdout__
       instead (which should also not be rebound). EmPy uses the standard Python error
       handlers when exceptions are raised in EmPy code, which print to sys.stderr.
     * The empy "module" exposed through the EmPy interface (e.g., @empy) is an artificial
       module. It cannot be imported with the import statement (and shouldn't -- it is an
       artifact of the EmPy processing system and does not correspond to any accessible .py
       file).
     * For an EmPy statement expansion all alone on a line, e.g., @{a = 1}, note that this
       will expand to a blank line due to the newline following the closing curly brace. To
       suppress this blank line, use the symmetric convention @{a = 1}@.
     * When using EmPy with make, note that partial output may be created before an error
       occurs; this is a standard caveat when using make. To avoid this, write to a temporary
       file and move when complete, delete the file in case of an error, use the -B option to
       fully buffer output (including the open), or (with GNU make) define a .DELETE_ON_ERROR
       target.
     * empy.identify tracks the context of executed EmPy code, not Python code. This means
       that blocks of code delimited with @{ and } will identify themselves as appearing on
       the line at which the } appears, and that pure Python code executed via the -D, -E and
       -F command line arguments will show up as all taking place on line 1. If you're
       tracking errors and want more information about the location of the errors from the
       Python code, use the -r command line option, which will provide you with the full
       Python traceback.

  Wish list

   Here are some random ideas for future revisions of EmPy. If any of these are of particular
   interest to you, your input would be appreciated.
     * Some real-world examples should really be included for demonstrating the power and
       expressiveness of EmPy first-hand.
     * A "trivial" mode, where all the EmPy system does is scan for simple tokens replace
       them with evaluations/executions, rather than having to do the contextual scanning it
       does now. This has the down side of being much less configurable and powerful but the
       upside of being extremely efficient. Perhaps this need not be a separate mode, but an
       additional prefix something of the form @<(...)>, @<{...}>, and possibly @<[12][...]>?
       Setting the trivial mode might simply disallow other expansions.
     * A "debug" mode, where EmPy prints the contents of everything it's about to evaluate
       (probably to stderr) before it does?
     * The ability to funnel all code through a configurable RExec for user-controlled
       security control. This would probably involve abstracting the execution functionality
       outside of the interpreter.
     * Optimized handling of processing would be nice for the possibility of an Apache module
       devoted to EmPy processing.
     * An EmPy emacs mode.
     * An "unbuffered" option which would lose contextual information like line numbers, but
       could potentially be more efficient at processing large files.
     * An optimization of offloading diversions to files when they become truly huge.
     * Unicode support, particularly for filters. (This may be problematic given Python 1.5.2
       support.)
     * Support for mapping filters (specified by dictionaries).
     * Support for some sort of batch processing, where several EmPy files can be listed at
       once and all of them evaluated with the same initial (presumably expensive)
       environment.
     * A more elaborate interactive mode, perhaps with a prompt and readline support.
     * A toplevel run function, which invoke delegates to, that accepts arguments similar to
       the command line as keyword arguments. Perhaps also a simplified wrapper just for
       doing basic processing, e.g., interpreter.simple?
     * A tool to collect significator information from a hierarchy of .em files and put them
       in a database form available for individual scripts would be extremely useful.
     * A StructuredText and/or reStructuredText filter would be quite useful, as would
       SGML/HTML/XML, s-expression, Python, etc. auto-indenter filters.
     * A caching system that stores off the compilations of repeated evaluations and
       executions so that in a persistent environment the same code does not have to be
       repeatedly evaluated/executed. This would probably be a necessity in an Apache
       module-based solution.
     * An option to change the format of the standard EmPy messages in a traceback.
     * An "binary" option to have EmPy process incoming data in chunks, rather than by lines,
       for handling of non-textual data or data which may not contain predictably short
       lines.
     * Support for some manner of implicitly processed /etc/empyrc and/or ~/.empyrc file, and
       of course an option to inhibit its processing. This can already be accomplished via an
       explicit EMPY_OPTIONS, but still ...
     * More uniform handling of the preprocessing directives (-I, -D, -E, -F, and -P),
       probably mapping directly to methods in the Interpreter class.
     * distutils support.

  Author's notes

   I originally conceived EmPy as a replacement for my [13]Web templating system which uses
   [14]m4 (a general macroprocessing system for UNIX).

   Most of my Web sites include a variety of m4 files, some of which are dynamically
   generated from databases, which are then scanned by a cataloging tool to organize them
   hierarchically (so that, say, a particular m4 file can understand where it is in the
   hierarchy, or what the titles of files related to it are without duplicating information);
   the results of the catalog are then written in database form as an m4 file (which every
   other m4 file implicitly includes), and then GNU make converts each m4 to an HTML file by
   processing it.

   As the Web sites got more complicated, the use of m4 (which I had originally enjoyed for
   the challenge and abstractness) really started to become an impediment to serious work;
   while I am very knowledgeable about m4 -- having used it for for so many years -- getting
   even simple things done with it is awkward and difficult. Worse yet, as I started to use
   Python more and more over the years, the cataloging programs which scanned the m4 and
   built m4 databases were migrated to Python and made almost trivial, but writing out huge
   awkward tables of m4 definitions simply to make them accessible in other m4 scripts
   started to become almost farcical -- especially when coupled with the difficulty in
   getting simple things done in m4.

   It occurred to me what I really wanted was an all-Python solution. But replacing what used
   to be the m4 files with standalone Python programs would result in somewhat awkward
   programs normally consisting mostly of unprocessed text punctuated by small portions where
   variables and small amounts of code need to be substituted. Thus the idea was a sort of
   inverse of a Python interpreter: a program that normally would just pass text through
   unmolested, but when it found a special signifier would execute Python code in a
   persistent environment. After considering between choices of signifiers, I settled on @
   and EmPy was born.

   As I developed the tool, I realized it could have general appeal, even to those with
   widely varying problems to solve, provided the core tool they needed was an interpreter
   that could embed Python code inside templated text. As I continue to use the tool, I have
   been adding features as unintrusively as possible as I see areas that can be improved.

   A design goal of EmPy is that its feature set should work on several levels; at each
   level, if the user does not wish or need to use features from another level, they are
   under no obligation to do so. If you have no need of substitutions, for instance, you are
   under no obligation to use them. If significators will not help you organize a set of EmPy
   scripts globally, then you need not use them. New features that are being added are
   whenever possible transparently backward compatible; if you do not need them, their
   introduction should not affect you in any way. The use of unknown prefix sequences results
   in errors, guaranteeing that they are reserved for future use.

  Release history

     * 2.3; 2003 Feb 20. Proper and full support for concurrent and recursive interpreters;
       protection from closing the true stdout file object; detect edge cases of interpreter
       globals or sys.stdout proxy collisions; add globals manipulation functions
       empy.getGlobals, empy.setGlobals, and empy.updateGlobals which properly preserve the
       empy pseudomodule; separate usage info out into easily accessible lists for easier
       presentation; have -h option show simple usage and -H show extened usage; add NullFile
       utility class.
     * 2.2.6; 2003 Jan 30. Fix a bug in the Filter.detach method (which would not normally be
       called anyway).
     * 2.2.5; 2003 Jan 9. Strip carriage returns out of executed code blocks for DOS/Windows
       compatibility.
     * 2.2.4; 2002 Dec 23. Abstract Filter interface to use methods only; add @[noop: ...]
       substitution for completeness and block commenting.
     * 2.2.3; 2002 Dec 16. Support compatibility with Jython by working around a minor
       difference between CPython and Jython in string splitting.
     * 2.2.2; 2002 Dec 14. Include better docstrings for pseudomodule functions; segue to a
       dictionary-based options system for interpreters; add empy.clearAllHooks and
       'empy.clearGlobals'; include a short documentation section on embedding interpreters;
       fix a bug in significator regular expression.
     * 2.2.1; 2002 Nov 30. Tweak test script to avoid writing unnecessary temporary file; add
       Interpreter.single method; expose evaluate, execute, substitute, and single methods to
       the pseudomodule; add (rather obvious) EMPY_OPTIONS environment variable support; add
       empy.enableHooks and 'empy.disableHooks'; include optimization to transparently
       disable hooks until they are actually used.
     * 2.2; 2002 Nov 21. Switched to -V option for version information; empy.createDiversion
       for creating initially empty diversion; direct access to diversion objects with
       'empy.retrieveDiversion'; environment variable support; removed --raw long argument
       (use --raw-errors instead); added quaternary escape code (well, why not).
     * 2.1; 2002 Oct 18. empy.atExit registry separate from hooks to allow for normal
       interpreter support; include a benchmark sample and test.sh verification script;
       expose empy.string directly; -D option for explicit defines on command line; remove
       ill-conceived support for @else: separator in @[if ...] substitution; handle nested
       substitutions properly; @[macro ...] substitution for creating recallable expansions.
     * 2.0.1; 2002 Oct 8. Fix missing usage information; fix after_evaluate hook not getting
       called; add empy.atExit call to register values.
     * 2.0; 2002 Sep 30. Parsing system completely revamped and simplified, eliminating a
       whole class of context-related bugs; builtin support for buffered filters; support for
       registering hooks; support for command line arguments; interactive mode with -i;
       significator value extended to be any valid Python expression.
     * 1.5.1; 2002 Sep 24. Allow @] to represent unbalanced close brackets in @[...] markups
       [now defunct; use escape codes instead].
     * 1.5; 2002 Sep 18. Escape codes (@\...); conditional and repeated expansion
       substitutions via @[if E:...], @[for X in E:...], and @[while E:...] notations; fix a
       few bugs involving files which do not end in newlines.
     * 1.4; 2002 Sep 7. Fix bug with triple quotes; collapse conditional and protected
       expression syntaxes into the single generalized @(...) notation; empy.setName and
       empy.setLine functions; true support for multiple concurrent interpreters with
       improved sys.stdout proxy; proper support for empy.expand to return a string evaluated
       in a subinterpreter as intended; merged Context and Parser classes together, and
       separated out Scanner functionality.
     * 1.3; 2002 Aug 24. Pseudomodule as true instance; move toward more verbose (and clear)
       pseudomodule functions; fleshed out diversion model; filters; conditional expressions;
       protected expressions; preprocessing with -P (in preparation for possible support for
       command line arguments).
     * 1.2; 2002 Aug 16. Treat bangpaths as comments; empy.quote for the opposite process of
       'empy.expand'; significators (@%... sequences); -I option; -f option; much improved
       documentation.
     * 1.1.5; 2002 Aug 15. Add a separate invoke function that can be called multiple times
       with arguments to simulate multiple runs.
     * 1.1.4; 2002 Aug 12. Handle strings thrown as exceptions properly; use getopt to
       process command line arguments; cleanup file buffering with AbstractFile; very slight
       documentation and code cleanup.
     * 1.1.3; 2002 Aug 9. Support for changing the prefix from within the empy pseudomodule.
     * 1.1.2; 2002 Aug 5. Renamed buffering option to -B, added -F option for interpreting
       Python files from the command line, fixed improper handling of exceptions from command
       line options (-E, -F).
     * 1.1.1; 2002 Aug 4. Typo bugfixes; documentation clarification.
     * 1.1; 2002 Aug 4. Added option for fully buffering output (including file opens),
       executing commands through the command line; some documentation errors fixed.
     * 1.0; 2002 Jul 23. Renamed project to EmPy. Documentation and sample tweaks; added
       empy.flatten. Added -a option.
     * 0.3; 2002 Apr 14. Extended "simple expression" syntax, interpreter abstraction, proper
       context handling, better error handling, explicit file inclusion, extended samples.
     * 0.2; 2002 Apr 13. Bugfixes, support non-expansion of Nones, allow choice of alternate
       prefix.
     * 0.1.1; 2002 Apr 12. Bugfixes, support for Python 1.5.x, add -r option.
     * 0.1; 2002 Apr 12. Initial early access release.

  Author

   This module was written by [15]Erik Max Francis. If you use this software, have
   suggestions for future releases, or bug reports, [16]I'd love to hear about it.

   Even if you try out EmPy for a project and find it unsuitable, I'd like to know what
   stumbling blocks you ran into so they can potentially be addressed in a future version.

  Version

   Version 2.3 $Date$ $Author$

   Modules and Packages
   [17]em

   A system for processing Python as markup embedded in text.
     _____________________________________________________________________________________

   [18]Table of Contents
   This document was automatically generated on Thu Feb 20 03:56:27 2003 by [19]HappyDoc
   version 2.0.1

References

   Visible links
   1. file://localhost/home/shafi/pak/empy-2.3/doc/index.html
   2. file://localhost/home/shafi/pak/empy-2.3/doc/index.html#refindex
   3. http://www.alcyone.com/pyos/empy/empy-latest.tar.gz
   4. http://www.alcyone.com/pyos/empy/
   5. http://www.gnu.org/copyleft/gpl.html
   6. mailto:empy-announce-list-subscribe@alcyone.com
   7. mailto:empy-list-subscribe@alcyone.com
   8. file://localhost/home/shafi/pak/empy-2.3/doc/index.html#refindex
   9. file://localhost/home/shafi/pak/empy-2.3/doc/index.html#refname
  10. file://localhost/home/shafi/pak/empy-2.3/doc/index.html#refsub
  11. file://localhost/home/shafi/pak/empy-2.3/doc/index.html#refi
  12. file://localhost/home/shafi/pak/empy-2.3/doc/index.html#ref...
  13. http://www.alcyone.com/max/info/m4.html
  14. http://www.seindal.dk/rene/gnu/
  15. http://www.alcyone.com/max/
  16. mailto:pyos@alcyone.com
  17. file://localhost/home/shafi/pak/empy-2.3/doc/em.py.html
  18. file://localhost/home/shafi/pak/empy-2.3/doc/index.html
  19. http://happydoc.sourceforge.net/

   Hidden links:
  20. file://localhost/home/shafi/pak/empy-2.3/doc/index.html#index