Tangle Source Files

A user initiates this process when they have a complete .w file that contains a description of source files. These source files are described with @o commands in the .w file.

The use case is successful when the source files are produced.

Outside this use case, the user will debug those source files, possibly updating the .w file. This will lead to a need to restart this use case.

The use case is a failure when the source files cannot be produced, due to errors in the .w file. These must be corrected based on information in log messages.

The sequence is simply ./pyweb.py *theFile*.w.

Weave Documentation

A user initiates this process when they have a .w file that contains a description of a document to produce. The document is described by the entire .w file.

The use case is successful when the documentation file is produced.

Outside this use case, the user will edit the documentation file, possibly updating the .w file. This will lead to a need to restart this use case.

The use case is a failure when the documentation file cannot be produced, due to errors in the .w file. These must be corrected based on information in log messages.

The sequence is simply ./pyweb.py *theFile*.w.

Tangle, Regression Test and Weave

A user initiates this process when they have a .w file that contains a description of a document to produce. The document is described by the entire .w file. Further, their final document should include regression test output from the source files created by the tangle operation.

The use case is successful when the documentation file is produced, including current regression test output.

Outside this use case, the user will edit the documentation file, possibly updating the .w file. This will lead to a need to restart this use case.

The use case is a failure when the documentation file cannot be produced, due to errors in the .w file. These must be corrected based on information in log messages.

The use case is a failure when the documentation file does not include current regression test output.

The sequence is as follows:

./pyweb.py -xw -pi theFile.w
python theTest >aLog
./pyweb.py -xt theFile.w

The first step excludes weaving and permits errors on the @i command. The -pi option is necessary in the event that the log file does not yet exist. The second step runs the regression test, creating a log file. The third step weaves the final document, including the regression test output.

Structural Tags

There are two definitional tags; these define the various chunks in an input file.

@o file @{ text @}

The @o (output) command defines a named output file chunk. The text is tangled to the named file with no alteration. It is woven into the document in an appropriate fixed-width font.

There are options available to specify comment conventions for the tangled output; this allows inclusion of source line numbers.

@d name @{ text @}

The @d (define) command defines a named chunk of program source. This text is tangled or woven when it is referenced by the reference inline tag.

There are options available to specify the indentation for this particular chunk. In rare cases, it can be helpful to override the indentation context.

Each @o and @d tag is followed by a chunk which is delimited by @{ and @} tags. At the end of that chunk, there is an optional "major" tag.

@|

A chunk may define user identifiers. The list of defined identifiers is placed in the chunk, separated by the @| separator.

Additionally, these tags provide for the inclusion of additional input files. This is necessary for decomposing a long document into easy-to-edit sections.

@i file

The @i (include) command includes another file. The previous chunk is ended. The file is processed completely, then a new chunk is started for the text after the @i command.

All material that is not explicitly in a @o or @d named chunk is implicitly collected into a sequence of anonymous document source chunks. These anonymous chunks form the backbone of the document that is woven. The anonymous chunks are never tangled into output program source files. They are woven into the document without any alteration.

Note that white space (line breaks ('\n'), tabs and spaces) have no effect on the input parsing. They are completely preserved on output.

The following example has three chunks:

Some RST-format documentation that describes the following piece of the
program.

@o myFile.py
@{
import math
print( math.pi )
@| math math.pi
@}

Some more RST documentation.

This starts with an anonymous chunk of documentation. It includes a named output chunk which will write to myFile.py. It ends with an anonymous chunk of documentation.

Inline Tags

There are several tags that are replaced by content in the woven output.

@@

The @@ command creates a single @ in the output file. This is replaced in tangled as well as woven output.

@<name@>

The name references a named chunk. When tangling, the referenced chunk replaces the reference command. When weaving, a reference marker is used. For example, in RST, this can be replaced with RST `reference`_ markup. Note that the indentation prior to the @< tag is preserved for the tangled chunk that replaces the tag.

@(Python expression@)

The Python expression is evaluated and the result is tangled or woven in place. A few global variables and modules are available. These are described in Expression Context.

Content Tags

There are three index creation tags that are replaced by content in the woven output.

@f

The @f command inserts a file cross reference. This lists the name of each file created by an @o command, and all of the various chunks that are concatenated to create this file.

@m

The @m command inserts a named chunk ("macro") cross reference. This lists the name of each chunk created by a @d command, and all of the various chunks that are concatenated to create the complete chunk.

@u

The @u command inserts a user identifier cross reference. This index lists the name of each chunk created by an @d command or @|, and all of the various chunks that are concatenated to create the complete chunk.

Additional Features

Sequence Numbers. The named chunks (from both @o and @d commands) are assigned unique sequence numbers to simplify cross references.

Case Sensitive. Chunk names and file names are case sensitive.

Abbreviations. Chunk names can be abbreviated. A partial name can have a trailing ellipsis (...), this will be resolved to the full name. The most typical use for this is shown in the following example:

Some RST-format documentation.

@o myFile.py
@{
@<imports of the various packages used>
print( math.pi,time.time() )
@}

Some notes on the packages used.

@d imports...
@{
import math,time
@| math time
@}

Some more RST-format documentation.

This example shows five chunks.

1. An anonymous chunk of documentation.
2. A named chunk that tangles the myFile.py output. It has a reference to the imports of the various packages used chunk. Note that the full name of the chunk is essentially a line of documentation, traditionally done as a comment line in a non-literate programming environment.
3. An anonymous chunk of documentation.
4. A named chunk with an abbreviated name. The imports... matches the name imports of the various packages used. Set off after the @| separator is the list of user-specified identifiers defined in this chunk.
5. An anonymous chunk of documentation.

Note that the first time a name appears (in a reference or definition), it must be the full name. All subsequent uses can be elisions. Also not that ambiguous elision is an annoying problem when you first start creating a document.

Concatenation. Named chunks are concatenated from their various pieces. This allows a named chunk to be broken into several pieces, simplifying the description. This is most often used when producing fairly complex output files.

An anonymous chunk with some RST documentation.

@o myFile.py
@{
import math,time
@}

Some notes on the packages used.

@o myFile.py
@{
print math.pi,time.time()
@}

Some more HTML documentation.

This example shows five chunks.

1. An anonymous chunk of documentation.
2. A named chunk that tangles the myFile.py output. It has the first part of the file. In the woven document this is marked with "=".
3. An anonymous chunk of documentation.
4. A named chunk that also tangles the myFile.py output. This chunk's content is appended to the first chunk. In the woven document this is marked with "+=".
5. An anonymous chunk of documentation.

Newline Preservation. Newline characters are preserved on input. Because of this the output may appear to have excessive newlines. In all of the above examples, each named chunk was defined with the following.

@{
import math,time
@}

This puts a newline character before and after the import line.

Controlling Indentation

We have two choices in indentation:

• Context-Sensitive.
• Consistent.

If we have context-sensitive indentation, then the indentation of a chunk reference is applied to the entire chunk when expanded in place of the reference. This makes it simpler to prepare source for languages (like Python) where indentation is important.

There are cases, however, when this is not desirable. There are some places in Python where we want to create long, triple-quoted strings with indentation that does not follow the prevailing indentations of the surrounding code.

Here's how the context-sensitive indentation works.

@o myFile.py
@{
def aFunction( a, b ):
    @<body of aFunction@>
@| aFunction @}

@d body...
@{
"""doc string"""
return a + b
@}

The tangled output from this will look like the following. All of the newline characters are preserved, and the reference to body of the aFunction is indented to match the prevailing indent where it was referenced. In the following example, explicit line markers of ~ are provided to make the blank lines more obvious.

~
~def aFunction( a, b ):
~
~    """doc string"""
~    return a + b
~

[The @| command shows that this chunk defines the identifier aFunction.]

This leads to a difficult design choice.

• Do we use context-sensitive indentation without any exceptions? This is the current implementation.

• Do we use consistent indentation and require the author to get it right? This seems to make Python awkward, since we might indent our outdent a @< name @> command, expecting the chunk to indent properly.

• Do we use context-sensitive indentation with an exception indicator? This seems to go against the utter simplicity we're cribbing from noweb. However, it makes a great deal of sense to add an option for @d chunks to supersede context-sensitive indentation. The author must then get it right.

  The syntax to define a section looks like this:

@d -noindent some chunk name
@{First partial line
More that uses """
@}

We might reference such a section like this.

@d some bigger chunk...
@{code
    @<some chunk name@>
@}

This will include the -noindent section by resetting the contextual indentation to zero. The First partial line line will be output after the four spaces provided by the some bigger chunk context.

After the first newline (More that uses """) will be at the left margin.

Tracking Source Line Numbers

Since the tangled output files are -- well -- tangled, it can be difficult to trace back from a Python error stack to the original line in the .w file that needs to be fixed.

To facilitate this, there is a two-step operation to get more detailed information on how tangling worked.

1. Use the -n command-line option to get line numbers.
2. Include comment indicators on the @o commands that define output files.

The expanded syntax for @o looks like this.

@o -start /* -end / page-layout.css
@{
*Some CSS code
@}

We've added two options: -start /* and -end */ which define comment start and end syntax. This will lead to comments embedded in the tangled output which contain source line numbers for every (every!) chunk.

Expression Context

There are two possible implementations for evaluation of a Python expression in the input.

1. Create an ExpressionCommand, and append this to the current Chunk. This will allow evaluation during weave processing and during tangle processing. This makes the entire weave (or tangle) context available to the expression, including completed cross reference information.
2. Evaluate the expression during input parsing, and append the resulting text as a TextCommand to the current Chunk. This provides a common result available to both weave and parse, but the only context available is the WebReader and the incomplete Web, built up to that point.

In this implementation, we adopt the latter approach, and evaluate expressions immediately. A simple global context is created with the following variables defined.

os.path:	This is the standard os.path module. The complete os module is not available. Just this one item.
datetime:	This is the standard datetime module.
platform:	This is the standard platform module.
__builtins__:	Most of the built-ins are available, too. Not all. exec(), eval(), open() and __import__() aren't available.
theLocation:	A tuple with the file name, first line number and last line number for the original expression's location.
theWebReader:	The WebReader instance doing the parsing.
theFile:	The .w file being processed.
thisApplication:
	The name of the running pyWeb application. It may not be pyweb.py, if some other script is being used.
__version__:	The version string in the pyWeb application.

./pyweb.py file...

Command Line Options

Currently, the following command line options are accepted.

-v:	Verbose logging.
-s:	Silent operation.
-cx:	Change the command character from @ to *x*.
-wweaver:	Choose a particular documentation weaver template. Currently the choices are RST and HTML.
-xw:	Exclude weaving. This does tangling of source program files only.
-xt:	Exclude tangling. This does weaving of the document file only.
-pcommand:	Permit errors in the given list of commands. The most common version is -pi to permit errors in locating an include file. This is done in the following scenario: pass 1 uses -xw -pi to exclude weaving and permit include-file errors; the tangled program is run to create test results; pass 2 uses -xt to exclude tangling and include the test results.

→Emitter superclass (3)
→Weaver subclass of Emitter to create documentation (12)
→RST subclass of Weaver (22)
→LaTeX subclass of Weaver (23)
→HTML subclass of Weaver (31), →(32)
→Tangler subclass of Emitter to create source files with no markup (43)
→TanglerMake subclass which is make-sensitive (48)

Emitter Superclass

The Emitter class is not a concrete class; it is never instantiated. It contains common features factored out of the Weaver and Tangler subclasses.

Inheriting from the Emitter class generally requires overriding one or more of the core methods: doOpen(), and doClose(). A subclass of Tangler, might override the code writing methods: quote(), codeBlock() or codeFinish().

The Emitter class defines the basic framework used to create and write to an output file. This class follows the Template design pattern. This design pattern directs us to factor the basic open(), close() and write() methods into two step algorithms.

def open( self ):
    common preparation
    self.doOpen() #overridden by subclasses
    return self

The common preparation section is generally internal housekeeping. The doOpen() method would be overridden by subclasses to change the basic behavior.

The class has the following attributes:

fileName:	the name of the current open file created by the open method
theFile:	the current open file created by the open method
linesWritten:	the total number of '\n' characters written to the file
totalFiles:	count of total number of files
totalLines:	count of total number of lines

Additionally, an emitter tracks an indentation context used by The codeBlock() method to indent each line written.

context:	the indentation context stack, updated by addIndent(), clrIndent() and readdIndent() methods.
lastIndent:	the last indent used after writing a line of source code
fragment:	the last line written was a fragment and needs a '\n'.
code_indent:	Any initial code indent. RST weavers needs additional code indentation. Other weavers don't care. Tanglers must have this set to zero.

Emitter superclass (3) =

class Emitter:
    """Emit an output file; handling indentation context."""
    code_indent= 0 # Used by a Tangler
    def __init__( self ):
        self.fileName= ""
        self.theFile= None
        self.linesWritten= 0
        self.totalFiles= 0
        self.totalLines= 0
        self.fragment= False
        self.logger= logging.getLogger( self.__class__.__qualname__ )
        self.log_indent= logging.getLogger( "indent." + self.__class__.__qualname__ )
        self.readdIndent( self.code_indent ) # Create context and initial lastIndent values
    def __str__( self ):
        return self.__class__.__name__
    →Emitter core open, close and write (4)
    →Emitter write a block of code (7), →(8), →(9)
    →Emitter indent control: set, clear and reset (10)

◊ Emitter superclass (3). Used by: Emitter class hierarchy... (2)

The core open() method tracks the open files. A subclass overrides a doOpen() method to name the output file, and then actually open the file. The Weaver will create an output file with a name that's based on the overall project. The Tangler will open the given file name.

The close() method closes the file. As with open(), a doClose() method actually closes the file. This allows subclasses to do overrides on the actual file processing.

The write() method is the lowest-level, unadorned write. This does some additional counting as well as writing the characters to the file.

Emitter core open, close and write (4) =

def open( self, aFile ):
    """Open a file."""
    self.fileName= aFile
    self.linesWritten= 0
    self.doOpen( aFile )
    return self
→Emitter doOpen, to be overridden by subclasses (5)
def close( self ):
    self.codeFinish() # Trailing newline for tangler only.
    self.doClose()
    self.totalFiles += 1
    self.totalLines += self.linesWritten
→Emitter doClose, to be overridden by subclasses (6)
def write( self, text ):
    if text is None: return
    self.linesWritten += text.count('\n')
    self.theFile.write( text )

# Context Manager
def __enter__( self ):
    return self
def __exit__( self, *exc ):
    self.close()
    return False

◊ Emitter core open, close and write (4). Used by: Emitter superclass (3)

The doOpen(), and doClose() methods are overridden by the various subclasses to perform the unique operation for the subclass.

Emitter doOpen, to be overridden by subclasses (5) =

def doOpen( self, aFile ):
    self.logger.debug( "creating {!r}".format(self.fileName) )

◊ Emitter doOpen, to be overridden by subclasses (5). Used by: Emitter core... (4)

Emitter doClose, to be overridden by subclasses (6) =

def doClose( self ):
    self.logger.debug( "wrote {:d} lines to {!s}".format(
        self.linesWritten, self.fileName) )

◊ Emitter doClose, to be overridden by subclasses (6). Used by: Emitter core... (4)

The codeBlock() method writes several lines of code. It calls the quote() method to quote each line of code after doing the correct indentation. Often, the last line of code is incomplete, so it is left unterminated. This last line of code also sets the indentation for any additional code to be tangled into this section.

The algorithm is as follows:

1. Save the topmost value of the context stack as the current indent.

2. Split the block of text on '\n' boundaries. There are two cases.

   • One line only, no newline.

     Write this with the saved lastIndent. The lastIndent is reset to zero since we've only written a fragmentary line.

   • Multiple lines.

     1. Write the first line with saved lastIndent.

     2. For each remaining line (except the last), write with the indented text, ending with a newline.

     3. The string split() method will put a trailing zero-length element in the list if the original block ended with a newline. We drop this zero length piece to prevent writing a useless fragment of indent-only after the final '\n'.

     4. If the last line has content: Write with the indented text, but do not write a trailing '\n'. Set lastIndent to zero because the next codeBlock() will continue this fragmentary line.

        If the last line has no content: Write nothing. Save the length of the last line as the most recent indent for any @<name@> reference to.

This feels a bit too complex. Indentation is a feature of a tangling a reference to a NamedChunk. It's not really a general feature of emitters or even tanglers.

Emitter write a block of code (7) =

def codeBlock( self, text ):
    """Indented write of a block of code. We buffer
    The spaces from the last line to act as the indent for the next line.
    """
    indent= self.context[-1]
    lines= text.split( '\n' )
    if len(lines) == 1: # Fragment with no newline.
        self.write('{!s}{!s}'.format(self.lastIndent*' ', lines[0]) )
        self.lastIndent= 0
        self.fragment= True
    else:
        first, rest= lines[:1], lines[1:]
        self.write('{!s}{!s}\n'.format(self.lastIndent*' ', first[0]) )
        for l in rest[:-1]:
            self.write( '{!s}{!s}\n'.format(indent*' ', l) )
        if rest[-1]:
            self.write( '{!s}{!s}'.format(indent*' ', rest[-1]) )
            self.lastIndent= 0
            self.fragment= True
        else:
            # Buffer a next indent
            self.lastIndent= len(rest[-1]) + indent
            self.fragment= False

◊ Emitter write a block of code (7). Used by: Emitter superclass (3)

The quote() method quotes a single line of source code. This is used by Weaver subclasses to transform source into a form acceptable by the final weave file format.

In the case of an HTML weaver, the HTML reserved characters -- <, >, &, and " -- must be replaced in the output of code with &lt;, &gt;, &amp;, and &quot;. However, since the author's original document sections contain HTML these will not be altered.

Emitter write a block of code (8) +=

quoted_chars = [
    # Must be empty for tangling.
]

def quote( self, aLine ):
    """Each individual line of code; often overridden by weavers to quote the code."""
    clean= aLine
    for from_, to_ in self.quoted_chars:
        clean= clean.replace( from_, to_ )
    return clean

◊ Emitter write a block of code (8). Used by: Emitter superclass (3)

The codeFinish() method handles a trailing fragmentary line when tangling.

Emitter write a block of code (9) +=

def codeFinish( self ):
    if self.fragment:
        self.write('\n')

◊ Emitter write a block of code (9). Used by: Emitter superclass (3)

These three methods are used when to be sure that the included text is indented correctly with respect to the surrounding text.

The addIndent() method pushes the next indent on the context stack using an increment to the previous indent.

When tangling, a "previous" value is set from the indent left over from the previous command. This allows @<name@> references to be indented properly. A tangle must track all nested @d contexts to create a proper global indent.

Weaving, however, is entirely localized to the block of code. There's no real context tracking. Just "lastIndent" from the previous command's codeBlock().

The setIndent() pushes a fixed indent instead adding an increment.

The clrIndent() method discards the most recent indent from the context stack. This is used when finished tangling a source chunk. This restores the indent to the prevailing indent.

The readdIndent() method removes all indent context information and resets the indent to a default.

Weaving may use an initial offset. It's an additional indent for woven code; not used for tangled code. In particular, RST requires this. readdIndent() uses this initial offset for weaving.

Emitter indent control: set, clear and reset (10) =

def addIndent( self, increment ):
    self.lastIndent= self.context[-1]+increment
    self.context.append( self.lastIndent )
    self.log_indent.debug( "addIndent {!s}: {!r}".format(increment, self.context) )
def setIndent( self, indent ):
    self.lastIndent= self.context[-1]
    self.context.append( indent )
    self.log_indent.debug( "setIndent {!s}: {!r}".format(indent, self.context) )
def clrIndent( self ):
    if len(self.context) > 1:
        self.context.pop()
    self.lastIndent= self.context[-1]
    self.log_indent.debug( "clrIndent {!r}".format(self.context) )
def readdIndent( self, indent=0 ):
    self.lastIndent= indent
    self.context= [self.lastIndent]
    self.log_indent.debug( "readdIndent {!s}: {!r}".format(indent, self.context) )

◊ Emitter indent control: set, clear and reset (10). Used by: Emitter superclass (3)

Weaver subclass of Emitter

A Weaver is an Emitter that produces the final user-focused document. This will include the source document with the code blocks surrounded by markup to present that code properly. In effect, the pyWeb @ commands are replaced by markup.

The Weaver class uses a simple set of templates to product RST markup as the default Subclasses can introduce other templates to produce HTML or LaTeX output.

Most Weaver languages don't rely on special indentation rules. The woven code samples usually start right on the left margin of the source document. However, the RST markup language does rely on extra indentation of code blocks. For that reason, the weavers have an additional indent for code blocks. This is generally set to zero, except when generating RST where 4 spaces is good.

The Weaver subclass extends an Emitter to weave the final documentation. This involves decorating source code to make it displayable. It also involves creating references and cross references among the various chunks.

The Weaver class adds several methods to the basic Emitter methods. These additional methods are also included that are used exclusively when weaving, never when tangling.

This class hierarch depends heavily on the string module.

Class-level variables include the following

extension:	The filename extension used by this weaver.
code_indent:	The number of spaces to indent code to separate code blocks from surrounding text. Mostly this is used by RST where a non-zero value is required.
header:	Any additional header material this weaver requires.

Instance-level configuration values:

reference_style:
	Either an instance of TransitiveReference() or SimpleReference()

Imports (11) =

import string

◊ Imports (11). Used by: pyweb.py (153)

Weaver subclass of Emitter to create documentation (12) =

class Weaver( Emitter ):
    """Format various types of XRef's and code blocks when weaving.
    RST format.
    Requires ``..  include:: <isoamsa.txt>``
    and      ``..  include:: <isopub.txt>``
    """
    extension= ".rst"
    code_indent= 4
    header= """\n..  include:: <isoamsa.txt>\n..  include:: <isopub.txt>\n"""

    def __init__( self ):
        super().__init__()
        self.reference_style= None # Must be configured.

    →Weaver doOpen, doClose and addIndent overrides (13)

    # Template Expansions.

    →Weaver quoted characters (14)
    →Weaver document chunk begin-end (15)
    →Weaver reference summary, used by code chunk and file chunk (16)
    →Weaver code chunk begin-end (17)
    →Weaver file chunk begin-end (18)
    →Weaver reference command output (19)
    →Weaver cross reference output methods (20), →(21)

◊ Weaver subclass of Emitter to create documentation (12). Used by: Emitter class hierarchy... (2)

The doOpen() method opens the file for writing. For weavers, the file extension is specified part of the target markup language being created.

The doClose() method extends the Emitter class close() method by closing the actual file created by the open() method.

The addIndent() reflects the fact that we're not tracking global indents, merely the local indentation required to weave a code chunk. The "indent" can vary because we're not always starting a fresh line with weaveReferenceTo().

Weaver doOpen, doClose and addIndent overrides (13) =

def doOpen( self, basename ):
    self.fileName= basename + self.extension
    self.logger.info( "Weaving {!r}".format(self.fileName) )
    self.theFile= open( self.fileName, "w" )
    self.readdIndent( self.code_indent )
def doClose( self ):
    self.theFile.close()
    self.logger.info( "Wrote {:d} lines to {!r}".format(
        self.linesWritten, self.fileName) )
def addIndent( self, increment=0 ):
    """increment not used when weaving"""
    self.context.append( self.context[-1] )
    self.log_indent.debug( "addIndent {!s}: {!r}".format(self.lastIndent, self.context) )
def codeFinish( self ):
    pass # Not needed when weaving

◊ Weaver doOpen, doClose and addIndent overrides (13). Used by: Weaver subclass of Emitter... (12)

This is an overly simplistic list. We use the parsed-literal directive because we're including links and what-not in the code. We have to quote certain inline markup -- but only when the characters are paired in a way that might confuse RST.

We really should use patterns like `.*?`, _.*?_, \*.*?\*, and \|.*?\| to look for paired RST inline markup and quote just these special character occurrences.

Weaver quoted characters (14) =

quoted_chars = [
    # prevent some RST markup from being recognized
    ('\\',r'\\'), # Must be first.
    ('`',r'\`'),
    ('_',r'\_'),
    ('*',r'\*'),
    ('|',r'\|'),
]

◊ Weaver quoted characters (14). Used by: Weaver subclass of Emitter... (12)

The remaining methods apply a chunk to a template.

The docBegin() and docEnd() methods are used when weaving a document text chunk. Typically, nothing is done before emitting these kinds of chunks. However, putting a .. line line number RST comment is an example of possible additional processing.

Weaver document chunk begin-end (15) =

def docBegin( self, aChunk ):
    pass
def docEnd( self, aChunk ):
    pass

◊ Weaver document chunk begin-end (15). Used by: Weaver subclass of Emitter... (12)

Each code chunk includes the places where the chunk is referenced.

Weaver reference summary, used by code chunk and file chunk (16) =

ref_template = string.Template( "${refList}" )
ref_separator = "; "
ref_item_template = string.Template( "$fullName (`${seq}`_)" )
def references( self, aChunk ):
    references= aChunk.references_list( self )
    if len(references) != 0:
        refList= [
            self.ref_item_template.substitute( seq=s, fullName=n )
            for n,s in references ]
        return self.ref_template.substitute( refList=self.ref_separator.join( refList ) )
    else:
        return ""

◊ Weaver reference summary, used by code chunk and file chunk (16). Used by: Weaver subclass of Emitter... (12)

The codeBegin() method emits the necessary material prior to a chunk of source code, defined with the @d command.

The codeEnd() method emits the necessary material subsequent to a chunk of source code, defined with the @d command. Links or cross references to chunks that refer to this chunk can be emitted.

Weaver code chunk begin-end (17) =

cb_template = string.Template( "\n..  _`${seq}`:\n..  rubric:: ${fullName} (${seq}) ${concat}\n..  parsed-literal::\n    :class: code\n\n" )

def codeBegin( self, aChunk ):
    txt = self.cb_template.substitute(
        seq= aChunk.seq,
        lineNumber= aChunk.lineNumber,
        fullName= aChunk.fullName,
        concat= "=" if aChunk.initial else "+=", # RST Separator
    )
    self.write( txt )

ce_template = string.Template( "\n..\n\n    ..  class:: small\n\n        |loz| *${fullName} (${seq})*. Used by: ${references}\n" )

def codeEnd( self, aChunk ):
    txt = self.ce_template.substitute(
        seq= aChunk.seq,
        lineNumber= aChunk.lineNumber,
        fullName= aChunk.fullName,
        references= self.references( aChunk ),
    )
    self.write(txt)

◊ Weaver code chunk begin-end (17). Used by: Weaver subclass of Emitter... (12)

The fileBegin() method emits the necessary material prior to a chunk of source code, defined with the @o command. A subclass would override this to provide specific text for the intended file type.

The fileEnd() method emits the necessary material subsequent to a chunk of source code, defined with the @o command.

There shouldn't be a list of references to a file. We assert that this list is always empty.

Weaver file chunk begin-end (18) =

fb_template = string.Template( "\n..  _`${seq}`:\n..  rubric:: ${fullName} (${seq}) ${concat}\n..  parsed-literal::\n    :class: code\n\n" )

def fileBegin( self, aChunk ):
    txt= self.fb_template.substitute(
        seq= aChunk.seq,
        lineNumber= aChunk.lineNumber,
        fullName= aChunk.fullName,
        concat= "=" if aChunk.initial else "+=", # RST Separator
    )
    self.write( txt )

fe_template= string.Template( "\n..\n\n    ..  class:: small\n\n        |loz| *${fullName} (${seq})*.\n" )

def fileEnd( self, aChunk ):
    assert len(self.references( aChunk )) == 0
    txt= self.fe_template.substitute(
        seq= aChunk.seq,
        lineNumber= aChunk.lineNumber,
        fullName= aChunk.fullName,
        references= [] )
    self.write( txt )

◊ Weaver file chunk begin-end (18). Used by: Weaver subclass of Emitter... (12)

The referenceTo() method emits a reference to a chunk of source code. There reference is made with a @<name@> reference within a @d or @o chunk. The references are defined with the @d or @o commands. A subclass would override this to provide specific text for the intended file type.

The referenceSep() method emits a separator to be used in a sequence of references. It's usually a ", ", but that might be changed to a simple " " because it looks better.

Weaver reference command output (19) =

refto_name_template= string.Template(r"|srarr|\ ${fullName} (`${seq}`_)")
refto_seq_template= string.Template("|srarr|\ (`${seq}`_)")
refto_seq_separator= ", "

def referenceTo( self, aName, seq ):
    """Weave a reference to a chunk.
    Provide name to get a full reference.
    name=None to get a short reference."""
    if aName:
        return self.refto_name_template.substitute( fullName= aName, seq= seq )
    else:
        return self.refto_seq_template.substitute( seq= seq )

def referenceSep( self ):
    """Separator between references."""
    return self.refto_seq_separator

◊ Weaver reference command output (19). Used by: Weaver subclass of Emitter... (12)

The xrefHead() method puts decoration in front of cross-reference output. A subclass may override this to change the look of the final woven document.

The xrefFoot() method puts decoration after cross-reference output. A subclass may override this to change the look of the final woven document.

The xrefLine() method is used for both file and chunk ("macro") cross-references to show a name (either file name or chunk name) and a list of chunks that reference the file or chunk.

The xrefDefLine() method is used for the user identifier cross-reference. This shows a name and a list of chunks that reference or define the name. One of the chunks is identified as the defining chunk, all others are referencing chunks.

An xrefEmpty() is used in the rare case of no user identifiers present.

The default behavior simply writes the Python data structure used to represent cross reference information. A subclass may override this to change the look of the final woven document.

Weaver cross reference output methods (20) =

xref_head_template = string.Template( "\n" )
xref_foot_template = string.Template( "\n" )
xref_item_template = string.Template( ":${fullName}:\n    ${refList}\n" )
xref_empty_template = string.Template( "(None)\n" )

def xrefHead( self ):
    txt = self.xref_head_template.substitute()
    self.write( txt )

def xrefFoot( self ):
    txt = self.xref_foot_template.substitute()
    self.write( txt )

def xrefLine( self, name, refList ):
    refList= [ self.referenceTo( None, r ) for r in refList ]
    txt= self.xref_item_template.substitute( fullName= name, refList = " ".join(refList) ) # RST Separator
    self.write( txt )

def xrefEmpty( self ):
    self.write( self.xref_empty_template.substitute() )

◊ Weaver cross reference output methods (20). Used by: Weaver subclass of Emitter... (12)

Cross-reference definition line

Weaver cross reference output methods (21) +=

name_def_template = string.Template( '[`${seq}`_]' )
name_ref_template = string.Template( '`${seq}`_' )

def xrefDefLine( self, name, defn, refList ):
    templates = { defn: self.name_def_template }
    refTxt= [ templates.get(r,self.name_ref_template).substitute( seq= r )
        for r in sorted( refList + [defn] )
        ]
    # Generic space separator
    txt= self.xref_item_template.substitute( fullName= name, refList = " ".join(refTxt) )
    self.write( txt )

◊ Weaver cross reference output methods (21). Used by: Weaver subclass of Emitter... (12)

RST subclass of Weaver

A degenerate case. This slightly simplifies the configuration and makes the output look a little nicer.

RST subclass of Weaver (22) =

class RST(Weaver):
    pass

◊ RST subclass of Weaver (22). Used by: Emitter class hierarchy... (2)

LaTeX subclass of Weaver

Experimental, at best.

An instance of LaTeX can be used by the Web object to weave an output document. The instance is created outside the Web, and given to the weave() method of the Web.

w= Web()
WebReader().load(w,"somefile.w")
weave_latex= LaTeX()
w.weave( weave_latex )

Note that the template language and LaTeX both use $. This means that all $ that are intended to be output to LaTeX must appear as $$ in the template.

The LaTeX subclass defines a Weaver that is customized to produce LaTeX output of code sections and cross reference information. Its markup is pretty rudimentary, but it's also distinctive enough to function pretty well in most L!sub:AT!sub:EX documents.

LaTeX subclass of Weaver (23) =

class LaTeX( Weaver ):
    """LaTeX formatting for XRef's and code blocks when weaving.
    Requires \\usepackage{fancyvrb}
    """
    extension= ".tex"
    code_indent= 0
    header= """\n\\usepackage{fancyvrb}\n"""

    →LaTeX code chunk begin (24)
    →LaTeX code chunk end (25)
    →LaTeX file output begin (26)
    →LaTeX file output end (27)
    →LaTeX references summary at the end of a chunk (28)
    →LaTeX write a line of code (29)
    →LaTeX reference to a chunk (30)

◊ LaTeX subclass of Weaver (23). Used by: Emitter class hierarchy... (2)

The LaTeX open() method opens the woven file by replacing the source file's suffix with ".tex" and creating the resulting file.

The LaTeX codeBegin() template writes the header prior to a chunk of source code. It aligns the block to the left, prints an italicised header, and opens a preformatted block.

LaTeX code chunk begin (24) =

cb_template = string.Template( """\\label{pyweb${seq}}
\\begin{flushleft}
\\textit{Code example ${fullName} (${seq})}
\\begin{Verbatim}[commandchars=\\\\\\{\\},codes={\\catcode`$$=3\\catcode`^=7},frame=single]\n""") # Prevent indent

◊ LaTeX code chunk begin (24). Used by: LaTeX subclass... (23)

The LaTeX codeEnd() template writes the trailer subsequent to a chunk of source code. This first closes the preformatted block and then calls the references() method to write a reference to the chunk that invokes this chunk; finally, it restores paragraph indentation.

LaTeX code chunk end (25) =

ce_template= string.Template("""
\\end{Verbatim}
${references}
\\end{flushleft}\n""") # Prevent indentation

◊ LaTeX code chunk end (25). Used by: LaTeX subclass... (23)

The LaTeX fileBegin() template writes the header prior to a the creation of a tangled file. Its formatting is identical to the start of a code chunk.

LaTeX file output begin (26) =

fb_template= cb_template

◊ LaTeX file output begin (26). Used by: LaTeX subclass... (23)

The LaTeX fileEnd() template writes the trailer subsequent to a tangled file. This closes the preformatted block, calls the LaTeX references() method to write a reference to the chunk that invokes this chunk, and restores normal indentation.

LaTeX file output end (27) =

fe_template= ce_template

◊ LaTeX file output end (27). Used by: LaTeX subclass... (23)

The references() template writes a list of references after a chunk of code. Each reference includes the example number, the title, and a reference to the LaTeX section and page numbers on which the referring block appears.

LaTeX references summary at the end of a chunk (28) =

ref_item_template = string.Template( """
\\item Code example ${fullName} (${seq}) (Sect. \\ref{pyweb${seq}}, p. \\pageref{pyweb${seq}})\n""")
ref_template = string.Template( """
\\footnotesize
Used by:
\\begin{list}{}{}
${refList}
\\end{list}
\\normalsize\n""")

◊ LaTeX references summary at the end of a chunk (28). Used by: LaTeX subclass... (23)

The quote() method quotes a single line of code to the weaver; since these lines are always in preformatted blocks, no special formatting is needed, except to avoid ending the preformatted block. Our one compromise is a thin space if the phrase \\end{Verbatim} is used in a code block.

LaTeX write a line of code (29) =

quoted_chars = [
    ("\\end{Verbatim}", "\\end\,{Verbatim}"), # Allow \end{Verbatim}
    ("\\{","\\\,{"), # Prevent unexpected commands in Verbatim
    ("$","\\$"), # Prevent unexpected math in Verbatim
]

◊ LaTeX write a line of code (29). Used by: LaTeX subclass... (23)

The referenceTo() template writes a reference to another chunk of code. It uses write directly as to follow the current indentation on the current line of code.

LaTeX reference to a chunk (30) =

refto_name_template= string.Template("""$$\\triangleright$$ Code Example ${fullName} (${seq})""")
refto_seq_template= string.Template("""(${seq})""")

◊ LaTeX reference to a chunk (30). Used by: LaTeX subclass... (23)

HTML subclasses of Weaver

This works, but, it's not clear that it should be kept.

An instance of HTML can be used by the Web object to weave an output document. The instance is created outside the Web, and given to the weave() method of the Web.

w= Web()
WebReader().load(w,"somefile.w")
weave_html= HTML()
w.weave( weave_html )

Variations in the output formatting are accomplished by having variant subclasses of HTML. In this implementation, we have two variations: full path references, and short references. The base class produces complete reference paths; a subclass produces abbreviated references.

The HTML subclass defines a Weaver that is customized to produce HTML output of code sections and cross reference information.

All HTML chunks are identified by anchor names of the form pyweb*n*. Each n is the unique chunk number, in sequential order.

An HTMLShort subclass defines a Weaver that produces HTML output with abbreviated (no name) cross references at the end of the chunk.

HTML subclass of Weaver (31) =

class HTML( Weaver ):
    """HTML formatting for XRef's and code blocks when weaving."""
    extension= ".html"
    code_indent= 0
    header= ""
    →HTML code chunk begin (33)
    →HTML code chunk end (34)
    →HTML output file begin (35)
    →HTML output file end (36)
    →HTML references summary at the end of a chunk (37)
    →HTML write a line of code (38)
    →HTML reference to a chunk (39)
    →HTML simple cross reference markup (40)

◊ HTML subclass of Weaver (31). Used by: Emitter class hierarchy... (2)

HTML subclass of Weaver (32) +=

class HTMLShort( HTML ):
    """HTML formatting for XRef's and code blocks when weaving with short references."""
    →HTML short references summary at the end of a chunk (42)

◊ HTML subclass of Weaver (32). Used by: Emitter class hierarchy... (2)

The codeBegin() template starts a chunk of code, defined with @d, providing a label and HTML tags necessary to set the code off visually.

HTML code chunk begin (33) =

cb_template= string.Template("""
<a name="pyweb${seq}"></a>
<!--line number ${lineNumber}-->
<p><em>${fullName}</em> (${seq})&nbsp;${concat}</p>
<code><pre>\n""")

◊ HTML code chunk begin (33). Used by: HTML subclass... (31)

The codeEnd() template ends a chunk of code, providing a HTML tags necessary to finish the code block visually. This calls the references method to write the list of chunks that reference this chunk.

HTML code chunk end (34) =

ce_template= string.Template("""
</pre></code>
<p>&loz; <em>${fullName}</em> (${seq}).
${references}
</p>\n""")

◊ HTML code chunk end (34). Used by: HTML subclass... (31)

The fileBegin() template starts a chunk of code, defined with @o, providing a label and HTML tags necessary to set the code off visually.

HTML output file begin (35) =

fb_template= string.Template("""<a name="pyweb${seq}"></a>
<!--line number ${lineNumber}-->
<p>``${fullName}`` (${seq})&nbsp;${concat}</p>
<code><pre>\n""") # Prevent indent

◊ HTML output file begin (35). Used by: HTML subclass... (31)

The fileEnd() template ends a chunk of code, providing a HTML tags necessary to finish the code block visually. This calls the references method to write the list of chunks that reference this chunk.

HTML output file end (36) =

fe_template= string.Template( """</pre></code>
<p>&loz; ``${fullName}`` (${seq}).
${references}
</p>\n""")

◊ HTML output file end (36). Used by: HTML subclass... (31)

The references() template writes the list of chunks that refer to this chunk. Note that this list could be rather long because of the possibility of transitive references.

HTML references summary at the end of a chunk (37) =

ref_item_template = string.Template(
'<a href="#pyweb${seq}"><em>${fullName}</em>&nbsp;(${seq})</a>'
)
ref_template = string.Template( '  Used by ${refList}.'  )

◊ HTML references summary at the end of a chunk (37). Used by: HTML subclass... (31)

The quote() method quotes an individual line of code for HTML purposes. This encodes the four basic HTML entities (<, >, &, ") to prevent code from being interpreted as HTML.

HTML write a line of code (38) =

quoted_chars = [
    ("&", "&"), # Must be first
    ("<", "&lt;"),
    (">", "&gt;"),
    ('"', "&quot;"),
]

◊ HTML write a line of code (38). Used by: HTML subclass... (31)

The referenceTo() template writes a reference to another chunk. It uses the direct write() method so that the reference is indented properly with the surrounding source code.

HTML reference to a chunk (39) =

refto_name_template = string.Template(
'<a href="#pyweb${seq}">&rarr;<em>${fullName}</em> (${seq})</a>'
)
refto_seq_template = string.Template(
'<a href="#pyweb${seq}">(${seq})</a>'
)

◊ HTML reference to a chunk (39). Used by: HTML subclass... (31)

The xrefHead() method writes the heading for any of the cross reference blocks created by @f, @m, or @u. In this implementation, the cross references are simply unordered lists.

The xrefFoot() method writes the footing for any of the cross reference blocks created by @f, @m, or @u. In this implementation, the cross references are simply unordered lists.

The xrefLine() method writes a line for the file or macro cross reference blocks created by @f or @m. In this implementation, the cross references are simply unordered lists.

HTML simple cross reference markup (40) =

xref_head_template = string.Template( "<dl>\n" )
xref_foot_template = string.Template( "</dl>\n" )
xref_item_template = string.Template( "<dt>${fullName}</dt><dd>${refList}</dd>\n" )
→HTML write user id cross reference line (41)

◊ HTML simple cross reference markup (40). Used by: HTML subclass... (31)

The xrefDefLine() method writes a line for the user identifier cross reference blocks created by @u. In this implementation, the cross references are simply unordered lists. The defining instance is included in the correct order with the other instances, but is bold and marked with a bullet (&bull;).

HTML write user id cross reference line (41) =

name_def_template = string.Template( '<a href="#pyweb${seq}"><b>&bull;${seq}</b></a>' )
name_ref_template = string.Template( '<a href="#pyweb${seq}">${seq}</a>' )

◊ HTML write user id cross reference line (41). Used by: HTML simple cross reference markup (40)

The HTMLShort subclass enhances the HTML class to provide short cross references. The references() method writes the list of chunks that refer to this chunk. Note that this list could be rather long because of the possibility of transitive references.

HTML short references summary at the end of a chunk (42) =

ref_item_template = string.Template( '<a href="#pyweb${seq}">(${seq})</a>' )

◊ HTML short references summary at the end of a chunk (42). Used by: HTML subclass... (32)

Tangler subclass of Emitter

The Tangler class is concrete, and can tangle source files. An instance of Tangler is given to the Web class tangle() method.

w= Web()
WebReader().load(w,"somefile.w")
t= Tangler()
w.tangle( t )

The Tangler subclass extends an Emitter to tangle the various program source files. The superclass is used to simply emit correctly indented source code and do very little else that could corrupt or alter the output.

Language-specific subclasses could be used to provide additional decoration. For example, inserting #line directives showing the line number in the original source file.

For Python, where indentation matters, the indent rules are relatively simple. The whitespace berfore a @< command is preserved as the prevailing indent for the block tangled as a replacement for the @<name@>.

There are three configurable values:

comment_start:	If not None, this is the leading character for a line-number comment
comment_end:	This is the trailing character for a line-number comment
include_line_numbers:
	Show the source line numbers in the output via additional comments.

Tangler subclass of Emitter to create source files with no markup (43) =

class Tangler( Emitter ):
    """Tangle output files."""
    def __init__( self ):
        super().__init__()
        self.comment_start= None
        self.comment_end= ""
        self.include_line_numbers= False
    →Tangler doOpen, and doClose overrides (44)
    →Tangler code chunk begin (45)
    →Tangler code chunk end (46)

◊ Tangler subclass of Emitter to create source files with no markup (43). Used by: Emitter class hierarchy... (2)

The default for all tanglers is to create the named file. In order to handle paths, we will examine the file name for any "/" characters and perform the required os.makedirs functions to allow creation of files with a path. We don't use Windows "\" characters, but rely on Python to handle this automatically.

This doClose() method overrides the Emitter class doClose() method by closing the actual file created by open.

Tangler doOpen, and doClose overrides (44) =

def checkPath( self ):
    if "/" in self.fileName:
        dirname, _, _ = self.fileName.rpartition("/")
        try:
            os.makedirs( dirname )
            self.logger.info( "Creating {!r}".format(dirname) )
        except OSError as e:
            # Already exists.  Could check for errno.EEXIST.
            self.logger.debug( "Exception {!r} creating {!r}".format(e, dirname) )
def doOpen( self, aFile ):
    self.fileName= aFile
    self.checkPath()
    self.theFile= open( aFile, "w" )
    self.logger.info( "Tangling {!r}".format(aFile) )
def doClose( self ):
    self.theFile.close()
    self.logger.info( "Wrote {:d} lines to {!r}".format(
        self.linesWritten, self.fileName) )

◊ Tangler doOpen, and doClose overrides (44). Used by: Tangler subclass of Emitter... (43)

The codeBegin() method starts emitting a new chunk of code. It does this by setting the Tangler's indent to the prevailing indent at the start of the @< reference command.

Tangler code chunk begin (45) =

def codeBegin( self, aChunk ):
    self.log_indent.debug( "<tangle {!s}:".format(aChunk.fullName) )
    if self.include_line_numbers and self.comment_start is not None:
        self.write( "\n{start!s} Web: {filename!s}:{line:d} {fullname!s}({seq:d}) {end!s}\n".format(
            start=  self.comment_start,
            webfilename= aChunk.web().webFileName,
            filename= aChunk.fileName,
            fullname= aChunk.fullName,
            name= aChunk.name,
            seq= aChunk.seq,
            line= aChunk.lineNumber,
            end= self.comment_end) )

◊ Tangler code chunk begin (45). Used by: Tangler subclass of Emitter... (43)

The codeEnd() method ends emitting a new chunk of code. It does this by resetting the Tangler's indent to the previous setting.

Tangler code chunk end (46) =

def codeEnd( self, aChunk ):
    self.log_indent.debug( ">{!s}".format(aChunk.fullName) )

◊ Tangler code chunk end (46). Used by: Tangler subclass of Emitter... (43)

TanglerMake subclass of Tangler

The TanglerMake class is can tangle source files. An instance of TanglerMake is given to the Web class tangle() method.

w= Web()
WebReader().load(w,"somefile.w")
t= TanglerMake()
w.tangle( t )

The TanglerMake subclass extends Tangler to make the source files more make-friendly. This subclass of Tangler does not touch an output file where there is no change. This is helpful when pyWeb's output is sent to make. Using TanglerMake assures that only files with real changes are rewritten, minimizing recompilation of an application for changes to the associated documentation.

This subclass of Tangler changes how files are opened and closed.

Imports (47) +=

import tempfile
import filecmp

◊ Imports (47). Used by: pyweb.py (153)

TanglerMake subclass which is make-sensitive (48) =

class TanglerMake( Tangler ):
    """Tangle output files, leaving files untouched if there are no changes."""
    def __init__( self, *args ):
        super().__init__( *args )
        self.tempname= None
    →TanglerMake doOpen override, using a temporary file (49)
    →TanglerMake doClose override, comparing temporary to original (50)

◊ TanglerMake subclass which is make-sensitive (48). Used by: Emitter class hierarchy... (2)

A TanglerMake creates a temporary file to collect the tangled output. When this file is completed, we can compare it with the original file in this directory, avoiding a "touch" if the new file is the same as the original.

TanglerMake doOpen override, using a temporary file (49) =

def doOpen( self, aFile ):
    fd, self.tempname= tempfile.mkstemp( dir=os.curdir )
    self.theFile= os.fdopen( fd, "w" )
    self.logger.info( "Tangling {!r}".format(aFile) )

◊ TanglerMake doOpen override, using a temporary file (49). Used by: TanglerMake subclass... (48)

If there is a previous file: compare the temporary file and the previous file.

If there was no previous file or the files are different: rename temporary to replace previous; else there was a previous file and the files were the same: unlink temporary and discard it.

This preserves the original (with the original date and time) if nothing has changed.

TanglerMake doClose override, comparing temporary to original (50) =

def doClose( self ):
    self.theFile.close()
    try:
        same= filecmp.cmp( self.tempname, self.fileName )
    except OSError as e:
        same= False # Doesn't exist.  Could check for errno.ENOENT
    if same:
        self.logger.info( "No change to {!r}".format(self.fileName) )
        os.remove( self.tempname )
    else:
        # Windows requires the original file name be removed first.
        self.checkPath()
        try:
            os.remove( self.fileName )
        except OSError as e:
            pass # Doesn't exist.  Could check for errno.ENOENT
        os.rename( self.tempname, self.fileName )
        self.logger.info( "Wrote {:d} lines to {!r}".format(
            self.linesWritten, self.fileName) )

◊ TanglerMake doClose override, comparing temporary to original (50). Used by: TanglerMake subclass... (48)

→Chunk class (52)
→NamedChunk class (63), →(68)
→OutputChunk class (69)
→NamedDocumentChunk class (73)

Chunk Superclass

An instance of the Chunk class has a life that includes four important events:

• creation,
• cross-reference,
• weave,
• and tangle.

A Chunk is created by a WebReader, and associated with a Web. There are several web append methods, depending on the exact subclass of Chunk. The WebReader calls the chunk's webAdd() method select the correct method for appending and indexing the chunk. Individual instances of Command are appended to the chunk. The basic outline for creating a Chunk instance is as follows:

w= Web( )
c= Chunk()
c.webAdd( w )
c.append( ...some Command... )
c.append( ...some Command... )

Before weaving or tangling, a cross reference is created for all user identifiers in all of the Chunk instances. This is done by: (1) visit each Chunk and call the getUserIDRefs() method to gather all identifiers; (2) for each identifier, visit each Chunk and call the searchForRE() method to find uses of the identifier.

ident= []
for c in the Web's named chunk list:
    ident.extend( c.getUserIDRefs() )
for i in ident:
    pattern= re.compile('W{!s}W'.format(i) )
    for c in the Web's named chunk list:
        c.searchForRE( pattern )

A Chunk is woven or tangled by the Web. The basic outline for weaving is as follows. The tangling action is essentially the same.

for c in the Web's chunk list:
    c.weave( aWeaver )

The Chunk class contains the overall definitions for all of the various specialized subclasses. In particular, it contains the append(), and appendText() methods used by all of the various Chunk subclasses.

When a @@ construct is located in the input stream, the stream contains three text tokens: material before the @@, the @@, and the material after the @@. These three tokens are reassembled into a single block of text. This reassembly is accomplished by changing the chunk's state so that the next TextCommand is appended onto the previous TextCommand.

The appendText() method either:

• appends to a previous TextCommand instance,
• or finds that there are no commands at all, and creates an initial TextCommand instance,
• or finds that the last Command isn't a subclass of TextCommand and creates a TextCommand instance.

Each subclass of Chunk has a particular type of text that it will process. Anonymous chunks only handle document text. The NamedChunk subclass that handles program source will override this method to create a different command type. The makeContent() method creates the appropriate Command instance for this Chunk subclass.

The weave() method of an anonymous Chunk uses the weaver's docBegin() and docEnd() methods to insert text that is source markup. Other subclasses will override this to use different Weaver methods for different kinds of text.

A Chunk has a Strategy object which is a subclass of Reference. This is either an instance of SimpleReference or TransitiveReference. A SimpleRerence does no additional processing, and locates the proximate reference to this chunk. The TransitiveReference walks "up" the web toward top-level file definitions that reference this Chunk.

The Chunk constructor initializes the following instance variables:

commands:	is a sequence of the various Command instances the comprise this chunk.
user_id_list:	is used the list of user identifiers associated with this chunk. This attribute is always None for this class. The NamedChunk subclass, however, can have user identifiers.
initial:	is True if this is the first definition (display with '=') or a subsequent definition (display with '+=').
web:	A weakref to the web which contains this Chunk. We want to inherit information from the Web overall.
fileName:	The file which contained this chunk's initial @o or @d.
name:	has the name of the chunk. This is '' for anonymous chunks.

!seq:

has the sequence number associated with this chunk. This is None for anonymous chunks.

referencedBy:	is the list of Chunks which reference this chunk.
references:	is the list of Chunks this chunk references.

Chunk class (52) =

class Chunk:
    """Anonymous piece of input file: will be output through the weaver only."""
    # construction and insertion into the web
    def __init__( self ):
        self.commands= [ ] # The list of children of this chunk
        self.user_id_list= None
        self.initial= None
        self.name= ''
        self.fullName= None
        self.seq= None
        self.fileName= ''
        self.referencedBy= [] # Chunks which reference this chunk.  Ideally just one.
        self.references= [] # Names that this chunk references

    def __str__( self ):
        return "\n".join( map( str, self.commands ) )
    def __repr__( self ):
        return "{!s}('{!s}')".format( self.__class__.__name__, self.name )
    →Chunk append a command (53)
    →Chunk append text (54)
    →Chunk add to the web (55)

    →Chunk generate references from this Chunk (58)
    →Chunk superclass make Content definition (56)
    →Chunk examination: starts with, matches pattern (57)
    →Chunk references to this Chunk (59)

    →Chunk weave this Chunk into the documentation (60)
    →Chunk tangle this Chunk into a code file (61)
    →Chunk indent adjustments (62)

◊ Chunk class (52). Used by: Chunk class hierarchy... (51)

The append() method simply appends a Command instance to this chunk.

Chunk append a command (53) =

def append( self, command ):
    """Add another Command to this chunk."""
    self.commands.append( command )
    command.chunk= self

◊ Chunk append a command (53). Used by: Chunk class (52)

The appendText() method appends a TextCommand to this chunk, or it concatenates it to the most recent TextCommand.

When an @@ construct is located, the appendText() method is used to accumulate this character. This means that it will be appended to any previous TextCommand, or new TextCommand will be built.

The reason for appending is that a TextCommand has an implicit indentation. The "@" cannot be a separate TextCommand because it will wind up indented.

Chunk append text (54) =

def appendText( self, text, lineNumber=0 ):
    """Append a single character to the most recent TextCommand."""
    try:
        # Works for TextCommand, otherwise breaks
        self.commands[-1].text += text
    except IndexError as e:
        # First command?  Then the list will have been empty.
        self.commands.append( self.makeContent(text,lineNumber) )
    except AttributeError as e:
        # Not a TextCommand?  Then there won't be a text attribute.
        self.commands.append( self.makeContent(text,lineNumber) )

◊ Chunk append text (54). Used by: Chunk class (52)

The webAdd() method adds this chunk to the given document web. Each subclass of the Chunk class must override this to be sure that the various Chunk subclasses are indexed properly. The Chunk class uses the add() method of the Web class to append an anonymous, unindexed chunk.

Chunk add to the web (55) =

def webAdd( self, web ):
    """Add self to a Web as anonymous chunk."""
    web.add( self )

◊ Chunk add to the web (55). Used by: Chunk class (52)

This superclass creates a specific Command for a given piece of content. A subclass can override this to change the underlying assumptions of that Chunk. The generic chunk doesn't contain code, it contains text and can only be woven, never tangled. A Named Chunk using @{ and @} creates code. A Named Chunk using @[ and @] creates text.

Chunk superclass make Content definition (56) =

def makeContent( self, text, lineNumber=0 ):
    return TextCommand( text, lineNumber )

◊ Chunk superclass make Content definition (56). Used by: Chunk class (52)

The startsWith() method examines a the first Command instance this Chunk instance to see if it starts with the given prefix string.

The lineNumber() method returns the line number of the first Command in this chunk. This provides some context for where the chunk occurs in the original input file.

A NamedChunk instance may define one or more identifiers. This parent class provides a dummy version of the getUserIDRefs method. The NamedChunk subclass overrides this to provide actual results. By providing this at the superclass-level, the Web can easily gather identifiers without knowing the actual subclass of Chunk.

The searchForRE() method examines each Command instance to see if it matches with the given regular expression. If so, this can be reported to the Web instance and accumulated as part of a cross reference for this Chunk.

Chunk examination: starts with, matches pattern (57) =

def startswith( self, prefix ):
    """Examine the first command's starting text."""
    return len(self.commands) >= 1 and self.commands[0].startswith( prefix )

def searchForRE( self, rePat ):
    """Visit each command, applying the pattern."""
    for c in self.commands:
        if c.searchForRE( rePat ):
            return self
    return None

@property
def lineNumber( self ):
    """Return the first command's line number or None."""
    return self.commands[0].lineNumber if len(self.commands) >= 1 else None

def getUserIDRefs( self ):
    return []

◊ Chunk examination: starts with, matches pattern (57). Used by: Chunk class (52)

The chunk search in the searchForRE() method parallels weaving and tangling a Chunk. The operation is delegated to each Command instance within the Chunk instance.

The genReferences() method visits each Command instance inside this chunk; a Command will yield the references.

Note that an exception may be raised by this operation if a referenced Chunk does not actually exist. If a reference Command does raise an error, we append this Chunk information and reraise the error with the additional context information.

Chunk generate references from this Chunk (58) =

def genReferences( self, aWeb ):
    """Generate references from this Chunk."""
    try:
        for t in self.commands:
            ref= t.ref( aWeb )
            if ref is not None:
                yield ref
    except Error as e:
        raise

◊ Chunk generate references from this Chunk (58). Used by: Chunk class (52)

The list of references to a Chunk uses a Strategy plug-in to either generate a simple parent or a transitive closure of all parents.

Note that we need to get the Weaver.reference_style which is a configuration item. This is a Strategy showing how to compute the list of references. The Weaver pushed it into the Web so that it is available for each Chunk.

Chunk references to this Chunk (59) =

def references_list( self, theWeaver ):
    """Extract name, sequence from Chunks into a list."""
    return [ (c.name, c.seq)
        for c in theWeaver.reference_style.chunkReferencedBy( self ) ]

◊ Chunk references to this Chunk (59). Used by: Chunk class (52)

The weave() method weaves this chunk into the final document as follows:

1. Call the Weaver class docBegin() method. This method does nothing for document content.
2. Visit each Command instance: call the Command instance weave() method to emit the content of the Command instance
3. Call the Weaver class docEnd() method. This method does nothing for document content.

Note that an exception may be raised by this action if a referenced Chunk does not actually exist. If a reference Command does raise an error, we append this Chunk information and reraise the error with the additional context information.

Chunk weave this Chunk into the documentation (60) =

def weave( self, aWeb, aWeaver ):
    """Create the nicely formatted document from an anonymous chunk."""
    aWeaver.docBegin( self )
    for cmd in self.commands:
        cmd.weave( aWeb, aWeaver )
    aWeaver.docEnd( self )
def weaveReferenceTo( self, aWeb, aWeaver ):
    """Create a reference to this chunk -- except for anonymous chunks."""
    raise Exception( "Cannot reference an anonymous chunk.""")
def weaveShortReferenceTo( self, aWeb, aWeaver ):
    """Create a short reference to this chunk -- except for anonymous chunks."""
    raise Exception( "Cannot reference an anonymous chunk.""")

◊ Chunk weave this Chunk into the documentation (60). Used by: Chunk class (52)

Anonymous chunks cannot be tangled. Any attempt indicates a serious problem with this program or the input file.

Chunk tangle this Chunk into a code file (61) =

def tangle( self, aWeb, aTangler ):
    """Create source code -- except anonymous chunks should not be tangled"""
    raise Error( 'Cannot tangle an anonymous chunk', self )

◊ Chunk tangle this Chunk into a code file (61). Used by: Chunk class (52)

Generally, a Chunk with a reference will adjust the indentation for that referenced material. However, this is not universally true, a subclass may not indent when tangling and may -- instead -- put stuff flush at the left margin by forcing the local indent to zero.

Chunk indent adjustments (62) =

def reference_indent( self, aWeb, aTangler, amount ):
    aTangler.addIndent( amount )  # Or possibly set indent to local zero.

def reference_dedent( self, aWeb, aTangler ):
    aTangler.clrIndent()

◊ Chunk indent adjustments (62). Used by: Chunk class (52)

NamedChunk class

A NamedChunk is created and used almost identically to an anonymous Chunk. The most significant difference is that a name is provided when the NamedChunk is created. This name is used by the Web to organize the chunks.

A NamedChunk is created with a @d or @o command. A NamedChunk contains programming language source when the brackets are @{ and @}. A separate subclass of NamedDocumentChunk is used when the brackets are @[ and @].

A NamedChunk can be both tangled into the output program files, and woven into the output document file.

The weave() method of a NamedChunk uses the Weaver's codeBegin() and codeEnd() methods to insert text that is program source and requires additional markup to make it stand out from documentation. Other subclasses can override this to use different Weaver methods for different kinds of text.

By inheritance from the superclass, this class indents. A separate subclass provides a no-indent implementation of a NamedChunk.

This class introduces some additional attributes.

fullName:	is the full name of the chunk. It's possible for a chunk to be an abbreviated forward reference; full names cannot be resolved until all chunks have been seen.
user_id_list:	is the list of user identifiers associated with this chunk.
refCount:	is the count of references to this chunk. If this is zero, the chunk is unused; if this is more than one, this chunk is multiply used. Either of these conditions is a possible error in the input. This is set by the usedBy() method.
name:	has the name of the chunk. Names can be abbreviated.

!seq:

has the sequence number associated with this chunk. This is set by the Web by the webAdd() method.

NamedChunk class (63) =

class NamedChunk( Chunk ):
    """Named piece of input file: will be output as both tangler and weaver."""
    def __init__( self, name ):
        super().__init__()
        self.name= name
        self.user_id_list= []
        self.refCount= 0
    def __str__( self ):
        return "{!r}: {!s}".format( self.name, Chunk.__str__(self) )
    def makeContent( self, text, lineNumber=0 ):
        return CodeCommand( text, lineNumber )
    →NamedChunk user identifiers set and get (64)
    →NamedChunk add to the web (65)
    →NamedChunk weave into the documentation (66)
    →NamedChunk tangle into the source file (67)

◊ NamedChunk class (63). Used by: Chunk class hierarchy... (51)

The setUserIDRefs() method accepts a list of user identifiers that are associated with this chunk. These are provided after the @| separator in a @d named chunk. These are used by the @u cross reference generator.

NamedChunk user identifiers set and get (64) =

def setUserIDRefs( self, text ):
    """Save user ID's associated with this chunk."""
    self.user_id_list= text.split()
def getUserIDRefs( self ):
    return self.user_id_list

◊ NamedChunk user identifiers set and get (64). Used by: NamedChunk class (63)

The webAdd() method adds this chunk to the given document Web instance. Each class of Chunk must override this to be sure that the various Chunk classes are indexed properly. This class uses the Web.addNamed() method of the Web class to append a named chunk.

NamedChunk add to the web (65) =

def webAdd( self, web ):
    """Add self to a Web as named chunk, update xrefs."""
    web.addNamed( self )

◊ NamedChunk add to the web (65). Used by: NamedChunk class (63)

The weave() method weaves this chunk into the final document as follows:

1. call the Weaver class codeBegin() method. This method emits the necessary markup for code appearing in the woven output.
2. visit each Command, calling the command's weave() method to emit the command's content.
3. call the Weaver class CodeEnd() method. This method emits the necessary markup for code appearing in the woven output.

For an RST weaver this becomes a parsed-literal, which requires a extra indent. For an HTML weaver this becomes a <pre> in a different-colored box.

References generate links in a woven document. In a tangled document, they create the actual code. The weaveRefenceTo() method weaves a reference to a chunk using both name and sequence number. The weaveShortReferenceTo() method weaves a reference to a chunk using only the sequence number. These references are created by ReferenceCommand instances within a chunk being woven.

The woven references simply follow whatever preceded them on the line; the indent (if any) doesn't change from the default.

NamedChunk weave into the documentation (66) =

def weave( self, aWeb, aWeaver ):
    """Create the nicely formatted document from a chunk of code."""
    self.fullName= aWeb.fullNameFor( self.name )
    aWeaver.addIndent()
    aWeaver.codeBegin( self )
    for cmd in self.commands:
        cmd.weave( aWeb, aWeaver )
    aWeaver.clrIndent( )
    aWeaver.codeEnd( self )
def weaveReferenceTo( self, aWeb, aWeaver ):
    """Create a reference to this chunk."""
    self.fullName= aWeb.fullNameFor( self.name )
    txt= aWeaver.referenceTo( self.fullName, self.seq )
    aWeaver.codeBlock( txt )
def weaveShortReferenceTo( self, aWeb, aWeaver ):
    """Create a shortened reference to this chunk."""
    txt= aWeaver.referenceTo( None, self.seq )
    aWeaver.codeBlock( txt )

◊ NamedChunk weave into the documentation (66). Used by: NamedChunk class (63)

The tangle() method tangles this chunk into the final document as follows:

1. call the Tangler class codeBegin() method to set indents properly.
2. visit each Command, calling the Command's tangle() method to emit the Command's content.
3. call the Tangler class codeEnd() method to restore indents.

If a ReferenceCommand does raise an error during tangling, we append this Chunk information and reraise the error with the additional context information.

NamedChunk tangle into the source file (67) =

def tangle( self, aWeb, aTangler ):
    """Create source code.
    Use aWeb to resolve @<namedChunk@>.
    Format as correctly indented source text
    """
    self.previous_command= TextCommand( "", self.commands[0].lineNumber )
    aTangler.codeBegin( self )
    for t in self.commands:
        try:
            t.tangle( aWeb, aTangler )
        except Error as e:
            raise
        self.previous_command= t
    aTangler.codeEnd( self )

◊ NamedChunk tangle into the source file (67). Used by: NamedChunk class (63)

There's a second variation on NamedChunk, one that doesn't indent based on context. It simply sets an indent at the left margin.

NamedChunk class (68) +=

class NamedChunk_Noindent( NamedChunk ):
    """Named piece of input file: will be output as both tangler and weaver."""
    def reference_indent( self, aWeb, aTangler, amount ):
        aTangler.setIndent( 0 )

    def reference_dedent( self, aWeb, aTangler ):
        aTangler.clrIndent()

◊ NamedChunk class (68). Used by: Chunk class hierarchy... (51)

OutputChunk class

A OutputChunk is created and used identically to a NamedChunk. The difference between this class and the parent class is the decoration of the markup when weaving.

The OutputChunk class is a subclass of NamedChunk that handles file output chunks defined with @o.

The weave() method of a OutputChunk uses the Weaver's fileBegin() and fileEnd() methods to insert text that is program source and requires additional markup to make it stand out from documentation. Other subclasses could override this to use different Weaver methods for different kinds of text.

All other methods, including the tangle method are identical to NamedChunk.

OutputChunk class (69) =

class OutputChunk( NamedChunk ):
    """Named piece of input file, defines an output tangle."""
    def __init__( self, name, comment_start=None, comment_end="" ):
        super().__init__( name )
        self.comment_start= comment_start
        self.comment_end= comment_end
    →OutputChunk add to the web (70)
    →OutputChunk weave (71)
    →OutputChunk tangle (72)

◊ OutputChunk class (69). Used by: Chunk class hierarchy... (51)

The webAdd() method adds this chunk to the given document Web. Each class of Chunk must override this to be sure that the various Chunk classes are indexed properly. This class uses the addOutput() method of the Web class to append a file output chunk.

OutputChunk add to the web (70) =

def webAdd( self, web ):
    """Add self to a Web as output chunk, update xrefs."""
    web.addOutput( self )

◊ OutputChunk add to the web (70). Used by: OutputChunk class (69)

The weave() method weaves this chunk into the final document as follows:

1. call the Weaver class codeBegin() method to emit proper markup for an output file chunk.
2. visit each Command, call the Command's weave() method to emit the Command's content.
3. call the Weaver class codeEnd() method to emit proper markup for an output file chunk.

These chunks of documentation are never tangled. Any attempt is an error.

If a ReferenceCommand does raise an error during weaving, we append this Chunk information and reraise the error with the additional context information.

OutputChunk weave (71) =

def weave( self, aWeb, aWeaver ):
    """Create the nicely formatted document from a chunk of code."""
    self.fullName= aWeb.fullNameFor( self.name )
    aWeaver.fileBegin( self )
    for cmd in self.commands:
        cmd.weave( aWeb, aWeaver )
    aWeaver.fileEnd( self )

◊ OutputChunk weave (71). Used by: OutputChunk class (69)

When we tangle, we provide the output Chunk's comment information to the Tangler to be sure that -- if line numbers were requested -- they can be included properly.

OutputChunk tangle (72) =

def tangle( self, aWeb, aTangler ):
    aTangler.comment_start= self.comment_start
    aTangler.comment_end= self.comment_end
    super().tangle( aWeb, aTangler )

◊ OutputChunk tangle (72). Used by: OutputChunk class (69)

NamedDocumentChunk class

A NamedDocumentChunk is created and used identically to a NamedChunk. The difference between this class and the parent class is that this chunk is only woven when referenced. The original definition is silently skipped.

The NamedDocumentChunk class is a subclass of NamedChunk that handles named chunks defined with @d and the @[...@] delimiters. These are woven slightly differently, since they are document source, not programming language source.

We're not as interested in the cross reference of named document chunks. They can be used multiple times or never. They are expected to be referenced by anonymous chunks. While this chunk subclass participates in this data gathering, it is ignored for reporting purposes.

All other methods, including the tangle method are identical to NamedChunk.

NamedDocumentChunk class (73) =

class NamedDocumentChunk( NamedChunk ):
    """Named piece of input file with document source, defines an output tangle."""
    def makeContent( self, text, lineNumber=0 ):
        return TextCommand( text, lineNumber )
    →NamedDocumentChunk weave (74)
    →NamedDocumentChunk tangle (75)

◊ NamedDocumentChunk class (73). Used by: Chunk class hierarchy... (51)

The weave() method quietly ignores this chunk in the document. A named document chunk is only included when it is referenced during weaving of another chunk (usually an anonymous document chunk).

The weaveReferenceTo() method inserts the content of this chunk into the output document. This is done in response to a ReferenceCommand in another chunk. The weaveShortReferenceTo() method calls the weaveReferenceTo() to insert the entire chunk.

NamedDocumentChunk weave (74) =

def weave( self, aWeb, aWeaver ):
    """Ignore this when producing the document."""
    pass
def weaveReferenceTo( self, aWeb, aWeaver ):
    """On a reference to this chunk, expand the body in place."""
    for cmd in self.commands:
        cmd.weave( aWeb, aWeaver )
def weaveShortReferenceTo( self, aWeb, aWeaver ):
    """On a reference to this chunk, expand the body in place."""
    self.weaveReferenceTo( aWeb, aWeaver )

◊ NamedDocumentChunk weave (74). Used by: NamedDocumentChunk class (73)

NamedDocumentChunk tangle (75) =

def tangle( self, aWeb, aTangler ):
    """Raise an exception on an attempt to tangle."""
    raise Error( "Cannot tangle a chunk defined with @[.""" )

◊ NamedDocumentChunk tangle (75). Used by: NamedDocumentChunk class (73)

→Command superclass (77)
→TextCommand class to contain a document text block (80)
→CodeCommand class to contain a program source code block (81)
→XrefCommand superclass for all cross-reference commands (82)
→FileXrefCommand class for an output file cross-reference (83)
→MacroXrefCommand class for a named chunk cross-reference (84)
→UserIdXrefCommand class for a user identifier cross-reference (85)
→ReferenceCommand class for chunk references (86)

Command Superclass

A Command is created by the WebReader, and attached to a Chunk. The Command participates in cross reference creation, weaving and tangling.

The Command superclass is abstract, and has default methods factored out of the various subclasses. When a subclass is created, it will override some of the methods provided in this superclass.

class MyNewCommand( Command ):
    ... overrides for various methods ...

Additionally, a subclass of WebReader must be defined to parse the new command syntax. The main process() function must also be updated to use this new subclass of WebReader.

The Command superclass provides the parent class definition for all of the various command types. The most common command is a block of text. For tangling it is presented with no changes. For weaving, quoting is used to make it work nicely with the given markup language.

The next most common command is a reference to a chunk. This is woven as a mark-up reference: a link. It is tangled as an expansion of the source code.

Additional methods include the following:

• The startswith() method examines any source text to see if it begins with the given prefix text.
• The searchForRE() method examines any source text to see if it matches the given regular expression, usually a match for a user identifier.
• The ref() method is ignored by all but the Reference subclass, which returns reference made by the command to the parent chunk.
• The weave() method weaves this into the output. If a document text command, it is emitted directly; if a program source code command, markup is applied. In the case of cross-reference commands, the actual cross-reference content is emitted. In the case of reference commands, they are woven as a reference to a named chunk.
• The tangle() method tangles this into the output. If a this is a document text command, it is ignored; if a this is a program source code command, it is indented and emitted. In the case of cross-reference commands, no output is produced. In the case of reference commands, the named chunk is indented and emitted.

The attributes of a Command instance includes the line number on which the command began, in lineNumber.

Command superclass (77) =

class Command:
    """A Command is the lowest level of granularity in the input stream."""
    def __init__( self, fromLine=0 ):
        self.lineNumber= fromLine+1 # tokenizer is zero-based
        self.chunk= None
        self.logger= logging.getLogger( self.__class__.__qualname__ )
    def __str__( self ):
        return "at {!r}".format(self.lineNumber)
    →Command analysis features: starts-with and Regular Expression search (78)
    →Command tangle and weave functions (79)

◊ Command superclass (77). Used by: Command class hierarchy... (76)

Command analysis features: starts-with and Regular Expression search (78) =

def startswith( self, prefix ):
    return None
def searchForRE( self, rePat ):
    return None
def indent( self ):
    return None

◊ Command analysis features: starts-with and Regular Expression search (78). Used by: Command superclass (77)

Command tangle and weave functions (79) =

def ref( self, aWeb ):
    return None
def weave( self, aWeb, aWeaver ):
    pass
def tangle( self, aWeb, aTangler ):
    pass

◊ Command tangle and weave functions (79). Used by: Command superclass (77)

TextCommand class

A TextCommand is created by a Chunk or a NamedDocumentChunk when a WebReader calls the chunk's appendText() method.

This Command participates in cross reference creation, weaving and tangling. When it is created, the source line number is provided so that this text can be tied back to the source document.

An instance of the TextCommand class is a block of document text. It can originate in an anonymous block or a named chunk delimited with @[ and @].

This subclass provides a concrete implementation for all of the methods. Since text is the author's original markup language, it is emitted directly to the weaver or tangler.

TextCommand class to contain a document text block (80) =

class TextCommand( Command ):
    """A piece of document source text."""
    def __init__( self, text, fromLine=0 ):
        super().__init__( fromLine )
        self.text= text
    def __str__( self ):
        return "at {!r}: {!r}...".format(self.lineNumber,self.text[:32])
    def startswith( self, prefix ):
        return self.text.startswith( prefix )
    def searchForRE( self, rePat ):
        return rePat.search( self.text )
    def indent( self ):
        if self.text.endswith('\n'):
            return 0
        try:
            last_line = self.text.splitlines()[-1]
            return len(last_line)
        except IndexError:
            return 0
    def weave( self, aWeb, aWeaver ):
        aWeaver.write( self.text )
    def tangle( self, aWeb, aTangler ):
        aTangler.write( self.text )

◊ TextCommand class to contain a document text block (80). Used by: Command class hierarchy... (76)

CodeCommand class

A CodeCommand is created by a NamedChunk when a WebReader calls the appendText() method. The Command participates in cross reference creation, weaving and tangling. When it is created, the source line number is provided so that this text can be tied back to the source document.

An instance of the CodeCommand class is a block of program source code text. It can originate in a named chunk (@d) with a @{ and @} delimiter. Or it can be a file output chunk (@o).

It uses the codeBlock() methods of a Weaver or Tangler. The weaver will insert appropriate markup for this code. The tangler will assure that the prevailing indentation is maintained.

CodeCommand class to contain a program source code block (81) =

class CodeCommand( TextCommand ):
    """A piece of program source code."""
    def weave( self, aWeb, aWeaver ):
        aWeaver.codeBlock( aWeaver.quote( self.text ) )
    def tangle( self, aWeb, aTangler ):
        aTangler.codeBlock( self.text )

◊ CodeCommand class to contain a program source code block (81). Used by: Command class hierarchy... (76)

XrefCommand superclass

An XrefCommand is created by the WebReader when any of the @f, @m, @u commands are found in the input stream. The Command is then appended to the current Chunk being built by the WebReader.

The XrefCommand superclass defines any common features of the various cross-reference commands (@f, @m, @u).

The formatXref() method creates the body of a cross-reference by the following algorithm:

1. Use the Weaver class xrefHead() method to emit the cross-reference header.
2. Sort the keys in the cross-reference mapping.
3. Use the Weaver class xrefLine() method to emit each line of the cross-reference mapping.
4. Use the Weaver class xrefFoot() method to emit the cross-reference footer.

If this command winds up in a tangle action, that use is illegal. An exception is raised and processing stops.

XrefCommand superclass for all cross-reference commands (82) =

class XrefCommand( Command ):
    """Any of the Xref-goes-here commands in the input."""
    def __str__( self ):
        return "at {!r}: cross reference".format(self.lineNumber)
    def formatXref( self, xref, aWeaver ):
        aWeaver.xrefHead()
        for n in sorted(xref):
            aWeaver.xrefLine( n, xref[n] )
        aWeaver.xrefFoot()
    def tangle( self, aWeb, aTangler ):
        raise Error('Illegal tangling of a cross reference command.')

◊ XrefCommand superclass for all cross-reference commands (82). Used by: Command class hierarchy... (76)

FileXrefCommand class

A FileXrefCommand is created by the WebReader when the @f command is found in the input stream. The Command is then appended to the current Chunk being built by the WebReader.

The FileXrefCommand class weave method gets the file cross reference from the overall web instance, and uses the formatXref() method of the XrefCommand superclass for format this result.

FileXrefCommand class for an output file cross-reference (83) =

class FileXrefCommand( XrefCommand ):
    """A FileXref command."""
    def weave( self, aWeb, aWeaver ):
        """Weave a File Xref from @o commands."""
        self.formatXref( aWeb.fileXref(), aWeaver )

◊ FileXrefCommand class for an output file cross-reference (83). Used by: Command class hierarchy... (76)

MacroXrefCommand class

A MacroXrefCommand is created by the WebReader when the @m command is found in the input stream. The Command is then appended to the current Chunk being built by the WebReader.

The MacroXrefCommand class weave method gets the named chunk (macro) cross reference from the overall web instance, and uses the formatXref() method of the XrefCommand superclass method for format this result.

MacroXrefCommand class for a named chunk cross-reference (84) =

class MacroXrefCommand( XrefCommand ):
    """A MacroXref command."""
    def weave( self, aWeb, aWeaver ):
        """Weave the Macro Xref from @d commands."""
        self.formatXref( aWeb.chunkXref(), aWeaver )

◊ MacroXrefCommand class for a named chunk cross-reference (84). Used by: Command class hierarchy... (76)

UserIdXrefCommand class

A MacroXrefCommand is created by the WebReader when the @u command is found in the input stream. The Command is then appended to the current Chunk being built by the WebReader.

The UserIdXrefCommand class weave method gets the user identifier cross reference information from the overall web instance. It then formats this line using the following algorithm, which is similar to the algorithm in the XrefCommand superclass.

1. Use the Weaver class xrefHead() method to emit the cross-reference header.
2. Sort the keys in the cross-reference mapping.
3. Use the Weaver class xrefDefLine() method to emit each line of the cross-reference definition mapping.
4. Use the Weaver class xrefFoor() method to emit the cross-reference footer.

UserIdXrefCommand class for a user identifier cross-reference (85) =

class UserIdXrefCommand( XrefCommand ):
    """A UserIdXref command."""
    def weave( self, aWeb, aWeaver ):
        """Weave a user identifier Xref from @d commands."""
        ux= aWeb.userNamesXref()
        if len(ux) != 0:
            aWeaver.xrefHead()
            for u in sorted(ux):
                defn, refList= ux[u]
                aWeaver.xrefDefLine( u, defn, refList )
            aWeaver.xrefFoot()
        else:
            aWeaver.xrefEmpty()

◊ UserIdXrefCommand class for a user identifier cross-reference (85). Used by: Command class hierarchy... (76)

ReferenceCommand class

A ReferenceCommand instance is created by a WebReader when a @<name@> construct in is found in the input stream. This is attached to the current Chunk being built by the WebReader.

During a weave, this creates a markup reference to another NamedChunk. During tangle, this actually includes the NamedChunk at this point in the tangled output file.

The constructor creates several attributes of an instance of a ReferenceCommand.

refTo:	the name of the chunk to which this refers, possibly elided with a trailing '...'.
fullName:	the full name of the chunk to which this refers.
chunkList:	the list of the chunks to which the name refers.

ReferenceCommand class for chunk references (86) =

class ReferenceCommand( Command ):
    """A reference to a named chunk, via @<name@>."""
    def __init__( self, refTo, fromLine=0 ):
        super().__init__( fromLine )
        self.refTo= refTo
        self.fullname= None
        self.sequenceList= None
        self.chunkList= []
    def __str__( self ):
        return "at {!r}: reference to chunk {!r}".format(self.lineNumber,self.refTo)
    →ReferenceCommand resolve a referenced chunk name (87)
    →ReferenceCommand refers to a chunk (88)
    →ReferenceCommand weave a reference to a chunk (89)
    →ReferenceCommand tangle a referenced chunk (90)

◊ ReferenceCommand class for chunk references (86). Used by: Command class hierarchy... (76)

The resolve() method queries the overall Web instance for the full name and sequence number for this chunk reference. This is used by the Weaver class referenceTo() method to write the markup reference to the chunk.

ReferenceCommand resolve a referenced chunk name (87) =

def resolve( self, aWeb ):
    """Expand our chunk name and list of parts"""
    self.fullName= aWeb.fullNameFor( self.refTo )
    self.chunkList= aWeb.getchunk( self.refTo )

◊ ReferenceCommand resolve a referenced chunk name (87). Used by: ReferenceCommand class... (86)

The ref() method is a request that is delegated by a Chunk; it resolves the reference this Command makes within the containing Chunk. When the Chunk iterates through the Commands, it can accumulate a list of Chinks to which it refers.

ReferenceCommand refers to a chunk (88) =

def ref( self, aWeb ):
    """Find and return the full name for this reference."""
    self.resolve( aWeb )
    return self.fullName

◊ ReferenceCommand refers to a chunk (88). Used by: ReferenceCommand class... (86)

The weave() method inserts a markup reference to a named chunk. It uses the Weaver class referenceTo() method to format this appropriately for the document type being woven.

ReferenceCommand weave a reference to a chunk (89) =

def weave( self, aWeb, aWeaver ):
    """Create the nicely formatted reference to a chunk of code."""
    self.resolve( aWeb )
    aWeb.weaveChunk( self.fullName, aWeaver )

◊ ReferenceCommand weave a reference to a chunk (89). Used by: ReferenceCommand class... (86)

The tangle() method inserts the resolved chunk in this place. When a chunk is tangled, it sets the indent, inserts the chunk and resets the indent.

This is where the Tangler indentation is updated by a reference. Or where indentation is set to a local zero because the included Chunk is a no-indent Chunk.

ReferenceCommand tangle a referenced chunk (90) =

def tangle( self, aWeb, aTangler ):
    """Create source code."""
    self.resolve( aWeb )

    self.logger.debug( "Indent {!r} + {!r}".format(aTangler.context, self.chunk.previous_command.indent()) )
    self.chunk.reference_indent( aWeb, aTangler, self.chunk.previous_command.indent() )

    self.logger.debug( "Tangling chunk {!r}".format(self.fullName) )
    if len(self.chunkList) != 0:
        for p in self.chunkList:
            p.tangle( aWeb, aTangler )
    else:
        raise Error( "Attempt to tangle an undefined Chunk, {!s}.".format( self.fullName, ) )

    self.chunk.reference_dedent( aWeb, aTangler )

◊ ReferenceCommand tangle a referenced chunk (90). Used by: ReferenceCommand class... (86)

Reference Superclass

The superclass is an abstract class that defines the interface for this object.

Reference class hierarchy - strategies for references to a chunk (91) =

class Reference:
    def __init__( self ):
        self.logger= logging.getLogger( self.__class__.__qualname__ )
    def chunkReferencedBy( self, aChunk ):
        """Return a list of Chunks."""
        pass

◊ Reference class hierarchy - strategies for references to a chunk (91). Used by: Base Class Definitions (1)

SimpleReference Class

The SimpleReference subclass does the simplest version of resolution. It returns the Chunks referenced.

Reference class hierarchy - strategies for references to a chunk (92) +=

class SimpleReference( Reference ):
    def chunkReferencedBy( self, aChunk ):
        refBy= aChunk.referencedBy
        return refBy

◊ Reference class hierarchy - strategies for references to a chunk (92). Used by: Base Class Definitions (1)

TransitiveReference Class

The TransitiveReference subclass does a transitive closure of all references to this Chunk.

This requires walking through the Web to locate "parents" of each referenced Chunk.

Reference class hierarchy - strategies for references to a chunk (93) +=

class TransitiveReference( Reference ):
    def chunkReferencedBy( self, aChunk ):
        refBy= aChunk.referencedBy
        self.logger.debug( "References: {!s}({:d}) {!r}".format(aChunk.name, aChunk.seq, refBy) )
        return self.allParentsOf( refBy )
    def allParentsOf( self, chunkList, depth=0 ):
        """Transitive closure of parents via recursive ascent.
        """
        final = []
        for c in chunkList:
            final.append( c )
            final.extend( self.allParentsOf( c.referencedBy, depth+1 ) )
        self.logger.debug( "References: {0:>{indent}s} {1!s}".format('--', final, indent=2*depth) )
        return final

◊ Reference class hierarchy - strategies for references to a chunk (93). Used by: Base Class Definitions (1)

class Web:
    """The overall Web of chunks."""
    def __init__( self ):
        self.webFileName= None
        self.chunkSeq= []
        self.output= {} # Map filename to Chunk
        self.named= {} # Map chunkname to Chunk
        self.sequence= 0
        self.logger= logging.getLogger( self.__class__.__qualname__ )
    def __str__( self ):
        return "Web {!r}".format( self.webFileName, )

    →Web construction methods used by Chunks and WebReader (97)
    →Web Chunk name resolution methods (102), →(103)
    →Web Chunk cross reference methods (104), →(106), →(107), →(108)
    →Web determination of the language from the first chunk (111)
    →Web tangle the output files (112)
    →Web weave the output document (113)

Web Construction

During web construction, it is convenient to capture information about the individual Chunk instances being appended to the web. This done using a Callback design pattern. Each subclass of Chunk provides an override for the Chunk class webAdd() method. This override calls one of the appropriate web construction methods.

Also note that the full name for a chunk can be given either as part of the definition, or as part a reference. Typically, the first reference has the full name and the definition has the elided name. This allows a reference to a chunk to contain a more complete description of the chunk.

We include a weakref to the Web to each Chunk.

Imports (96) +=

import weakref

◊ Imports (96). Used by: pyweb.py (153)

Web construction methods used by Chunks and WebReader (97) =

→Web add full chunk names, ignoring abbreviated names (98)
→Web add an anonymous chunk (99)
→Web add a named macro chunk (100)
→Web add an output file definition chunk (101)

◊ Web construction methods used by Chunks and WebReader (97). Used by: Web class... (95)

A name is only added to the known names when it is a full name, not an abbreviation ending with "...". Abbreviated names are quietly skipped until the full name is seen.

The algorithm for the addDefName() method, then is as follows:

1. Use the fullNameFor() method to locate the full name.
2. If no full name was found (the result of fullNameFor() ends with '...'), ignore this name as an abbreviation with no definition.
3. If this is a full name and the name was not in the named mapping, add this full name to the mapping.

This name resolution approach presents a problem when a chunk's first definition uses an abbreviated name.

Web add full chunk names, ignoring abbreviated names (98) =

def addDefName( self, name ):
    """Reference to or definition of a chunk name."""
    nm= self.fullNameFor( name )
    if nm is None: return None
    if nm[-3:] == '...':
        self.logger.debug( "Abbreviated reference {!r}".format(name) )
        return None # first occurance is a forward reference using an abbreviation
    if nm not in self.named:
        self.named[nm]= []
        self.logger.debug( "Adding empty chunk {!r}".format(name) )
    return nm

◊ Web add full chunk names, ignoring abbreviated names (98). Used by: Web construction... (97)

An anonymous Chunk is kept in a sequence of Chunks, used for tangling.

Web add an anonymous chunk (99) =

def add( self, chunk ):
    """Add an anonymous chunk."""
    self.chunkSeq.append( chunk )
    chunk.web= weakref.ref(self)

◊ Web add an anonymous chunk (99). Used by: Web construction... (97)

A named Chunk is defined with a @d command. It is collected into a mapping of NamedChunk instances. An entry in the mapping is a sequence of chunks that have the same name. This sequence of chunks is used to produce the weave or tangle output.

All chunks are also placed in the overall sequence of chunks. This overall sequence is used for weaving the document.

The addDefName() method is used to resolve this name if it is an abbreviation, or add it to the mapping if this is the first occurance of the name. If the name cannot be added, an instance of our Error class is raised. If the name exists or was added, the chunk is appended to the chunk list associated with this name.

The Web's sequence counter is incremented, and this unique sequence number sets the seq attribute of the Chunk. If the chunk list was empty, this is the first chunk, the initial flag is set to True when there's only one element in the list. Otherwise, it's False.

Web add a named macro chunk (100) =

def addNamed( self, chunk ):
    """Add a named chunk to a sequence with a given name."""
    self.chunkSeq.append( chunk )
    chunk.web= weakref.ref(self)
    nm= self.addDefName( chunk.name )
    if nm:
        # We found the full name for this chunk
        self.sequence += 1
        chunk.seq= self.sequence
        chunk.fullName= nm
        self.named[nm].append( chunk )
        chunk.initial= len(self.named[nm]) == 1
        self.logger.debug( "Extending chunk {!r} from {!r}".format(nm, chunk.name) )
    else:
        raise Error("No full name for {!r}".format(chunk.name), chunk)

◊ Web add a named macro chunk (100). Used by: Web construction... (97)

An output file definition Chunk is defined with an @o command. It is collected into a mapping of OutputChunk instances. An entry in the mapping is a sequence of chunks that have the same name. This sequence of chunks is used to produce the weave or tangle output.

Note that file names cannot be abbreviated.

All chunks are also placed in overall sequence of chunks. This overall sequence is used for weaving the document.

If the name does not exist in the output mapping, the name is added with an empty sequence of chunks. In all cases, the chunk is appended to the chunk list associated with this name.

The web's sequence counter is incremented, and this unique sequence number sets the Chunk's seq attribute. If the chunk list was empty, this is the first chunk, the initial flag is True if this is the first chunk.

Web add an output file definition chunk (101) =

def addOutput( self, chunk ):
    """Add an output chunk to a sequence with a given name."""
    self.chunkSeq.append( chunk )
    chunk.web= weakref.ref(self)
    if chunk.name not in self.output:
        self.output[chunk.name] = []
        self.logger.debug( "Adding chunk {!r}".format(chunk.name) )
    self.sequence += 1
    chunk.seq= self.sequence
    chunk.fullName= chunk.name
    self.output[chunk.name].append( chunk )
    chunk.initial = len(self.output[chunk.name]) == 1

◊ Web add an output file definition chunk (101). Used by: Web construction... (97)

Web Chunk Name Resolution

Web Chunk name resolution has three aspects. The first is resolving elided names (those ending with ...) to their full names. The second is finding the named chunk in the web structure. The third is returning a reference to a specific chunk including the name and sequence number.

Note that a Chunk name actually refers to a sequence of Chunk instances. Multiple definitions for a Chunk are allowed, and all of the definitions are concatenated to create the complete Chunk. This complexity makes it unwise to return the sequence of same-named Chunk; therefore, we put the burden on the Web to process all Chunk with a given name, in sequence.

The fullNameFor() method resolves full name for a chunk as follows:

1. If the string is already in the named mapping, this is the full name
2. If the string ends in '...', visit each key in the dictionary to see if the key starts with the string up to the trailing '...'. If a match is found, the dictionary key is the full name.
3. Otherwise, treat this as a full name.

Web Chunk name resolution methods (102) =

def fullNameFor( self, name ):
    """Resolve "..." names into the full name."""
    if name in self.named: return name
    if name[-3:] == '...':
        best= [ n for n in self.named.keys()
            if n.startswith( name[:-3] ) ]
        if len(best) > 1:
            raise Error("Ambiguous abbreviation {!r}, matches {!r}".format( name, list(sorted(best)) ) )
        elif len(best) == 1:
            return best[0]
    return name

◊ Web Chunk name resolution methods (102). Used by: Web class... (95)

The getchunk() method locates a named sequence of chunks by first determining the full name for the identifying string. If full name is in the named mapping, the sequence of chunks is returned. Otherwise, an instance of our Error class is raised because the name is unresolvable.

It might be more helpful for debugging to emit this as an error in the weave and tangle results and keep processing. This would allow an author to catch multiple errors in a single run of pyWeb.

Web Chunk name resolution methods (103) +=

def getchunk( self, name ):
    """Locate a named sequence of chunks."""
    nm= self.fullNameFor( name )
    if nm in self.named:
        return self.named[nm]
    raise Error( "Cannot resolve {!r} in {!r}".format(name,self.named.keys()) )

◊ Web Chunk name resolution methods (103). Used by: Web class... (95)

Web Cross-Reference Support

Cross-reference support includes creating and reporting on the various cross-references available in a web. This includes creating the list of chunks that reference a given chunk; and returning the file, macro and user identifier cross references.

Each Chunk has a list Reference commands that shows the chunks to which a chunk refers. These relationships must be reversed to show the chunks that refer to a given chunk. This is done by traversing the entire web of named chunks and recording each chunk-to-chunk reference. This mapping has the referred-to chunk as the key, and a sequence of referring chunks as the value.

The accumulation is initiated by the web's createUsedBy() method. This method visits a Chunk, calling the genReferences() method, passing in the Web instance as an argument. Each Chunk class genReferences() method, in turn, invokes the usedBy() method of each Command instance in the chunk. Most commands do nothing, but a ReferenceCommand will resolve the name to which it refers.

When the createUsedBy() method has accumulated the entire cross reference, it also assures that all chunks are used exactly once.

Web Chunk cross reference methods (104) =

def createUsedBy( self ):
    """Update every piece of a Chunk to show how the chunk is referenced.
    Each piece can then report where it's used in the web.
    """
    for aChunk in self.chunkSeq:
        #usage = (self.fullNameFor(aChunk.name), aChunk.seq)
        for aRefName in aChunk.genReferences( self ):
            for c in self.getchunk( aRefName ):
                c.referencedBy.append( aChunk )
                c.refCount += 1
    →Web Chunk check reference counts are all one (105)

◊ Web Chunk cross reference methods (104). Used by: Web class... (95)

We verify that the reference count for a Chunk is exactly one. We don't gracefully tolerate multiple references to a Chunk or unreferenced chunks.

Web Chunk check reference counts are all one (105) =

for nm in self.no_reference():
    self.logger.warn( "No reference to {!r}".format(nm) )
for nm in self.multi_reference():
    self.logger.warn( "Multiple references to {!r}".format(nm) )
for nm in self.no_definition():
    self.logger.error( "No definition for {!r}".format(nm) )
    self.errors += 1

◊ Web Chunk check reference counts are all one (105). Used by: Web Chunk cross reference methods... (104)

The one-pass version

for nm,cl in self.named.items():
    if len(cl) > 0:
        if cl[0].refCount == 0:
           self.logger.warn( "No reference to {!r}".format(nm) )
        elif cl[0].refCount > 1:
           self.logger.warn( "Multiple references to {!r}".format(nm) )
    else:
        self.logger.error( "No definition for {!r}".format(nm) )

We use three methods to filter chunk names into the various warning categories. The no_reference list is a list of chunks defined by never referenced. The multi_reference list is a list of chunks defined by never referenced. The no_definition list is a list of chunks referenced but not defined.

Web Chunk cross reference methods (106) +=

def no_reference( self ):
    return [ nm for nm,cl in self.named.items() if len(cl)>0 and cl[0].refCount == 0 ]
def multi_reference( self ):
    return [ nm for nm,cl in self.named.items() if len(cl)>0 and cl[0].refCount > 1 ]
def no_definition( self ):
    return [ nm for nm,cl in self.named.items() if len(cl) == 0 ]

◊ Web Chunk cross reference methods (106). Used by: Web class... (95)

The fileXref() method visits all named file output chunks in output and collects the sequence numbers of each section in the sequence of chunks.

The chunkXref() method uses the same algorithm as a the fileXref() method, but applies it to the named mapping.

Web Chunk cross reference methods (107) +=

def fileXref( self ):
    fx= {}
    for f,cList in self.output.items():
        fx[f]= [ c.seq for c in cList ]
    return fx
def chunkXref( self ):
    mx= {}
    for n,cList in self.named.items():
        mx[n]= [ c.seq for c in cList ]
    return mx

◊ Web Chunk cross reference methods (107). Used by: Web class... (95)

The userNamesXref() method creates a mapping for each user identifier. The value for this mapping is a tuple with the chunk that defined the identifer (via a @| command), and a sequence of chunks that reference the identifier.

For example: { 'Web': ( 87, (88,93,96,101,102,104) ), 'Chunk': ( 53, (54,55,56,60,57,58,59) ) }, shows that the identifier 'Web' is defined in chunk with a sequence number of 87, and referenced in the sequence of chunks that follow.

This works in two passes:

1. _gatherUserId() gathers all user identifiers
2. _updateUserId() searches all text commands for the identifiers and updates the Web class cross reference information.

Web Chunk cross reference methods (108) +=

def userNamesXref( self ):
    ux= {}
    self._gatherUserId( self.named, ux )
    self._gatherUserId( self.output, ux )
    self._updateUserId( self.named, ux )
    self._updateUserId( self.output, ux )
    return ux
def _gatherUserId( self, chunkMap, ux ):
    →collect all user identifiers from a given map into ux (109)
def _updateUserId( self, chunkMap, ux ):
    →find user identifier usage and update ux from the given map (110)

◊ Web Chunk cross reference methods (108). Used by: Web class... (95)

User identifiers are collected by visiting each of the sequence of Chunks that share the same name; within each component chunk, if chunk has identifiers assigned by the @| command, these are seeded into the dictionary. If the chunk does not permit identifiers, it simply returns an empty list as a default action.

collect all user identifiers from a given map into ux (109) =

for n,cList in chunkMap.items():
    for c in cList:
        for id in c.getUserIDRefs():
            ux[id]= ( c.seq, [] )

◊ collect all user identifiers from a given map into ux (109). Used by: Web Chunk cross reference methods... (108)

User identifiers are cross-referenced by visiting each of the sequence of Chunks that share the same name; within each component chunk, visit each user identifier; if the Chunk class searchForRE() method matches an identifier, this is appended to the sequence of chunks that reference the original user identifier.

find user identifier usage and update ux from the given map (110) =

# examine source for occurrences of all names in ux.keys()
for id in ux.keys():
    self.logger.debug( "References to {!r}".format(id) )
    idpat= re.compile( r'\W{!s}\W'.format(id) )
    for n,cList in chunkMap.items():
        for c in cList:
            if c.seq != ux[id][0] and c.searchForRE( idpat ):
                ux[id][1].append( c.seq )

◊ find user identifier usage and update ux from the given map (110). Used by: Web Chunk cross reference methods... (108)

Loop Detection

How do we assure that the web is a proper tree and doesn't contain any loops?

Consider this example web

@o example1 @{
    @<part 1A@>
@}

@d part 1A @{
    @<part 1B@>
@}

@d part 1B @{
    @<part 1A@>
@}

All valid chunks are must be referenced from a @o chunk, either directly, or indirectly via one or more @<name@> references. This defines a proper tree with @o at the root and children at each @d.

Each chunk can have multiple references to further @d definitions. No chunk can reference the @o definition at the root.

To be circular, two @d chunks must reference each other.

To be valid, either (or both) must be named by the @o. There will, therefore, be two references: from the @o and a @d. Our check for duplicate references will spot this.

We do not need to do a proper BFS or DFS through the graph to check for loops. The simple reference count will do.

Tangle and Weave Support

The language() method makes a stab at determining the output language. The determination of the language can be done a variety of ways. One is to use command line parameters, another is to use the filename extension on the input file.

We examine the first few characters of input. A proper HTML, XHTML or XML file begins with '<!', '<?' or '<H'. LaTeX files typically begin with '%' or ''. Everything else is probably RST.

Web determination of the language from the first chunk (111) =

def language( self, preferredWeaverClass=None ):
    """Construct a weaver appropriate to the document's language"""
    if preferredWeaverClass:
        return preferredWeaverClass()
    self.logger.debug( "Picking a weaver based on first chunk {!r}".format(self.chunkSeq[0][:4]) )
    if self.chunkSeq[0].startswith('<'):
        return HTML()
    if self.chunkSeq[0].startswith('%') or self.chunkSeq[0].startswith('\\'):
        return LaTeX()
    return RST()

◊ Web determination of the language from the first chunk (111). Used by: Web class... (95)

The tangle() method of the Web class performs the tangle() method for each Chunk of each named output file. Note that several Chunks may share the file name, requiring the file be composed of material from each Chunk, in order.

Web tangle the output files (112) =

def tangle( self, aTangler ):
    for f, c in self.output.items():
        with aTangler.open(f):
            for p in c:
                p.tangle( self, aTangler )

◊ Web tangle the output files (112). Used by: Web class... (95)

The weave() method of the Web class creates the final documentation. This is done by stepping through each Chunk in sequence and weaving the chunk into the resulting file via the Chunk class weave() method.

During weaving of a chunk, the chunk may reference another chunk. When weaving a reference to a named chunk (output or ordinary programming source defined with @{), this does not lead to transitive weaving: only a reference is put in from one chunk to another. However, when weaving a chunk defined with @[, the chunk is expanded when weaving. The decision is delegated to the referenced chunk.

TODO Can we refactor weaveChunk out of here entirely?

Should it go in ReferenceCommand weave...?

Web weave the output document (113) =

def weave( self, aWeaver ):
    self.logger.debug( "Weaving file from {!r}".format(self.webFileName) )
    basename, _ = os.path.splitext( self.webFileName )
    with aWeaver.open(basename):
        for c in self.chunkSeq:
            c.weave( self, aWeaver )
def weaveChunk( self, name, aWeaver ):
    self.logger.debug( "Weaving chunk {!r}".format(name) )
    chunkList= self.getchunk(name)
    if not chunkList:
        raise Error( "No Definition for {!r}".format(name) )
    chunkList[0].weaveReferenceTo( self, aWeaver )
    for p in chunkList[1:]:
        aWeaver.write( aWeaver.referenceSep() )
        p.weaveShortReferenceTo( self, aWeaver )

◊ Web weave the output document (113). Used by: Web class... (95)

The WebReader Class

There are two forms of the constructor for a WebReader. The initial WebReader instance is created with code like the following:

p= WebReader()
p.command = options.commandCharacter

This will define the command character; usually provided as a command-line parameter to the application.

When processing an include file (with the @i command), a child WebReader instance is created with code like the following:

c= WebReader( parent=parentWebReader )

This will inherit the configuration from the parent WebReader. This will also include a reference from child to parent so that embedded Python expressions can view the entire input context.

The WebReader class parses the input file into command blocks. These are assembled into Chunks, and the Chunks are assembled into the document Web. Once this input pass is complete, the resulting Web can be tangled or woven.

"Structural" commands define the structure of the Chunks. The structural commands are @d and @o, as well as the @{, @}, @[, @] brackets, and the @i command to include another file.

"Inline" commands are inline within a Chunk: they define internal Commands. Blocks of text are minor commands, as well as the @<name@> references. The @@ escape is also handled here so that all further processing is independent of any parsing.

"Content" commands generate woven content. These include the various cross-reference commands (@f, @m and @u).

There are two class-level OptionParser instances used by this class.

output_option_parser:
	An OptionParser used to parse the @o command.
definition_option_parser:
	An OptionParser used to parse the @d command.

The class has the following attributes:

parent:	is the outer WebReader when processing a @i command.
command:	is the command character; a WebReader will use the parent command character if the parent is not None.
permitList:	is the list of commands that are permitted to fail. This is generally an empty list or ('@i',).
_source:	The open source being used by load().
fileName:	is used to pass the file name to the Web instance.
theWeb:	is the current open Web.
tokenizer:	An instance of Tokenizer used to parse the input. This is built when load() is called.
aChunk:	is the current open Chunk being built.
totalLines:
totalFiles:	Summaries

WebReader class - parses the input file, building the Web structure (114) =

class WebReader:
    """Parse an input file, creating Chunks and Commands."""

    output_option_parser= OptionParser(
        OptionDef( "-start", nargs=1, default=None ),
        OptionDef( "-end", nargs=1, default="" ),
        OptionDef( "argument", nargs='*' ),
        )

    definition_option_parser= OptionParser(
        OptionDef( "-indent", nargs=0 ),
        OptionDef( "-noindent", nargs=0 ),
        OptionDef( "argument", nargs='*' ),
        )

    def __init__( self, parent=None ):
        self.logger= logging.getLogger( self.__class__.__qualname__ )

        # Configuration of this reader.
        self.parent= parent
        if self.parent:
            self.command= self.parent.command
            self.permitList= self.parent.permitList
        else: # Defaults until overridden
            self.command= '@'
            self.permitList= []

        # Load options
        self._source= None
        self.fileName= None
        self.theWeb= None

        # State of reading and parsing.
        self.tokenizer= None
        self.aChunk= None

        # Summary
        self.totalLines= 0
        self.totalFiles= 0
        self.errors= 0

        →WebReader command literals (130)
    def __str__( self ):
        return self.__class__.__name__
    →WebReader location in the input stream (128)
    →WebReader load the web (129)
    →WebReader handle a command string (115), →(127)

◊ WebReader class - parses the input file, building the Web structure (114). Used by: Base Class Definitions (1)

Command recognition is done via a Chain of Command-like design. There are two conditions: the command string is recognized or it is not recognized. If the command is recognized, handleCommand() either:

• (for "structural" commands) attaches the current Chunk (self.aChunk) to the current Web (self.aWeb), or
• (for "inline" and "content" commands) create a Command, attach it to the current Chunk (self.aChunk)

and returns a true result.

If the command is not recognized, handleCommand() returns false.

A subclass can override handleCommand() to

1. call this superclass version;
2. if the command is unknown to the superclass, then the subclass can attempt to process it;
3. if the command is unknown to both classes, then return false. Either a subclass will handle it, or the default activity taken by load() is to treat the command a text, but also issue a warning.

WebReader handle a command string (115) =

def handleCommand( self, token ):
    self.logger.debug( "Reading {!r}".format(token) )
    →major commands segment the input into separate Chunks (116)
    →minor commands add Commands to the current Chunk (121)
    elif token[:2] in (self.cmdlcurl,self.cmdlbrak):
        # These should have been consumed as part of @o and @d parsing
        self.logger.error( "Extra {!r} (possibly missing chunk name) near {!r}".format(token, self.location()) )
        self.errors += 1
    else:
        return None # did not recogize the command
    return True # did recognize the command

◊ WebReader handle a command string (115). Used by: WebReader class... (114)

The following sequence of if-elif statements identifies the structural commands that partition the input into separate Chunks.

major commands segment the input into separate Chunks (116) =

if token[:2] == self.cmdo:
    →start an OutputChunk, adding it to the web (117)
elif token[:2] == self.cmdd:
    →start a NamedChunk or NamedDocumentChunk, adding it to the web (118)
elif token[:2] == self.cmdi:
    →import another file (119)
elif token[:2] in (self.cmdrcurl,self.cmdrbrak):
    →finish a chunk, start a new Chunk adding it to the web (120)

◊ major commands segment the input into separate Chunks (116). Used by: WebReader handle a command... (115)

An output chunk has the form @o name @{ content @}. We use the first two tokens to name the OutputChunk. We simply expect the @{ separator. We then attach all subsequent commands to this chunk while waiting for the final @} token to end the chunk.

We'll use an OptionParser to locate the optional parameters. This will then let us build an appropriate instance of OutputChunk.

With some small additional changes, we could use OutputChunk( **options ).

start an OutputChunk, adding it to the web (117) =

args= next(self.tokenizer)
self.expect( (self.cmdlcurl,) )
options= self.output_option_parser.parse( args )
self.aChunk= OutputChunk( name=options['argument'],
        comment_start= options.get('start',None),
        comment_end= options.get('end',""),
        )
self.aChunk.fileName= self.fileName
self.aChunk.webAdd( self.theWeb )
# capture an OutputChunk up to @}

◊ start an OutputChunk, adding it to the web (117). Used by: major commands... (116)

A named chunk has the form @d name @{ content @} for code and @d name @[ content @] for document source. We use the first two tokens to name the NamedChunk or NamedDocumentChunk. We expect either the @{ or @[ separator, and use the actual token found to choose which subclass of Chunk to create. We then attach all subsequent commands to this chunk while waiting for the final @} or @] token to end the chunk.

We'll use an OptionParser to locate the optional parameter of -noindent.

[Or possibly -indent number?]

Then we can use options to create an appropriate subclass of NamedChunk.

If "-indent" is in options, this is the default. If both are in the options, we can provide a warning, I guess.

TODO Add a warning for conflicting options.

start a NamedChunk or NamedDocumentChunk, adding it to the web (118) =

args= next(self.tokenizer)
brack= self.expect( (self.cmdlcurl,self.cmdlbrak) )
options= self.output_option_parser.parse( args )
name=options['argument']

if brack == self.cmdlbrak:
    self.aChunk= NamedDocumentChunk( name )
elif brack == self.cmdlcurl:
    if '-noindent' in options:
        self.aChunk= NamedChunk_Noindent( name )
    else:
        self.aChunk= NamedChunk( name )
elif brack == None:
    pass # Error noted by expect()
else:
    raise Error( "Design Error" )

self.aChunk.fileName= self.fileName
self.aChunk.webAdd( self.theWeb )
# capture a NamedChunk up to @} or @]

◊ start a NamedChunk or NamedDocumentChunk, adding it to the web (118). Used by: major commands... (116)

An import command has the unusual form of @i name, with no trailing separator. When we encounter the @i token, the next token will start with the file name, but may continue with an anonymous chunk. We require that all @i commands occur at the end of a line, and break on the '\n' which must occur after the file name. This permits file names with embedded spaces. It also permits arguments and options, if really necessary.

Once we have split the file name away from the rest of the following anonymous chunk, we push the following token back into the token stream, so that it will be the first token examined at the top of the load() loop.

We create a child WebReader instance to process the included file. The entire file is loaded into the current Web instance. A new, empty Chunk is created at the end of the file so that processing can resume with an anonymous Chunk.

The reader has a permitList attribute. This lists any commands where failure is permitted. Currently, only the @i command can be set to permit failure; this allows a .w to include a file that does not yet exist.

The primary use case for this feature is when weaving test output. The first pass of pyWeb tangles the program source files; they are then run to create test output; the second pass of pyWeb weaves this test output into the final document via the @i command.

import another file (119) =

incFile= next(self.tokenizer).strip()
try:
    self.logger.info( "Including {!r}".format(incFile) )
    include= WebReader( parent=self )
    include.load( self.theWeb, incFile )
    self.totalLines += include.tokenizer.lineNumber
    self.totalFiles += include.totalFiles
    if include.errors:
        self.errors += include.errors
        self.logger.error(
            "Errors in included file {!s}, output is incomplete.".format(
            incFile) )
except Error as e:
    self.logger.error(
        "Problems with included file {!s}, output is incomplete.".format(
        incFile) )
    self.errors += 1
except IOError as e:
    self.logger.error(
        "Problems with included file {!s}, output is incomplete.".format(
        incFile) )
    # Discretionary -- sometimes we want to continue
    if self.cmdi in self.permitList: pass
    else: raise # TODO: Seems heavy-handed
self.aChunk= Chunk()
self.aChunk.webAdd( self.theWeb )

◊ import another file (119). Used by: major commands... (116)

When a @} or @] are found, this finishes a named chunk. The next text is therefore part of an anonymous chunk.

Note that no check is made to assure that the previous Chunk was indeed a named chunk or output chunk started with @{ or @[. To do this, an attribute would be needed for each Chunk subclass that indicated if a trailing bracket was necessary. For the base Chunk class, this would be false, but for all other subclasses of Chunk, this would be true.

finish a chunk, start a new Chunk adding it to the web (120) =

self.aChunk= Chunk()
self.aChunk.webAdd( self.theWeb )

◊ finish a chunk, start a new Chunk adding it to the web (120). Used by: major commands... (116)

The following sequence of elif statements identifies the minor commands that add Command instances to the current open Chunk.

minor commands add Commands to the current Chunk (121) =

elif token[:2] == self.cmdpipe:
    →assign user identifiers to the current chunk (122)
elif token[:2] == self.cmdf:
    self.aChunk.append( FileXrefCommand(self.tokenizer.lineNumber) )
elif token[:2] == self.cmdm:
    self.aChunk.append( MacroXrefCommand(self.tokenizer.lineNumber) )
elif token[:2] == self.cmdu:
    self.aChunk.append( UserIdXrefCommand(self.tokenizer.lineNumber) )
elif token[:2] == self.cmdlangl:
    →add a reference command to the current chunk (123)
elif token[:2] == self.cmdlexpr:
    →add an expression command to the current chunk (125)
elif token[:2] == self.cmdcmd:
    →double at-sign replacement, append this character to previous TextCommand (126)

◊ minor commands add Commands to the current Chunk (121). Used by: WebReader handle a command... (115)

User identifiers occur after a @| in a NamedChunk.

Note that no check is made to assure that the previous Chunk was indeed a named chunk or output chunk started with @{. To do this, an attribute would be needed for each Chunk subclass that indicated if user identifiers are permitted. For the base Chunk class, this would be false, but for the NamedChunk class and OutputChunk class, this would be true.

User identifiers are name references at the end of a NamedChunk These are accumulated and expanded by @u reference

assign user identifiers to the current chunk (122) =

try:
    self.aChunk.setUserIDRefs( next(self.tokenizer).strip() )
except AttributeError:
    # Out of place @| user identifier command
    self.logger.error( "Unexpected references near {!s}: {!s}".format(self.location(),token) )
    self.errors += 1

◊ assign user identifiers to the current chunk (122). Used by: minor commands... (121)

A reference command has the form @<name@>. We accept three tokens from the input, the middle token is the referenced name.

add a reference command to the current chunk (123) =

# get the name, introduce into the named Chunk dictionary
expand= next(self.tokenizer).strip()
closing= self.expect( (self.cmdrangl,) )
self.theWeb.addDefName( expand )
self.aChunk.append( ReferenceCommand( expand, self.tokenizer.lineNumber ) )
self.aChunk.appendText( "", self.tokenizer.lineNumber ) # to collect following text
self.logger.debug( "Reading {!r} {!r}".format(expand, closing) )

◊ add a reference command to the current chunk (123). Used by: minor commands... (121)

An expression command has the form @(Python Expression@). We accept three tokens from the input, the middle token is the expression.

There are two alternative semantics for an embedded expression.

• Deferred Execution. This requires definition of a new subclass of Command, ExpressionCommand, and appends it into the current Chunk. At weave and tangle time, this expression is evaluated. The insert might look something like this: aChunk.append( ExpressionCommand(expression, self.tokenizer.lineNumber) ).
• Immediate Execution. This simply creates a context and evaluates the Python expression. The output from the expression becomes a TextCommand, and is append to the current Chunk.

We use the Immediate Execution semantics.

Note that we've removed the blanket os. We only provide os.path. An os.getcwd() must be changed to os.path.realpath('.').

Imports (124) +=

import builtins
import sys
import platform

◊ Imports (124). Used by: pyweb.py (153)

add an expression command to the current chunk (125) =

# get the Python expression, create the expression result
expression= next(self.tokenizer)
self.expect( (self.cmdrexpr,) )
try:
    # Build Context
    safe= types.SimpleNamespace( **dict( (name,obj)
        for name,obj in builtins.__dict__.items()
        if name not in ('eval', 'exec', 'open', '__import__')))
    globals= dict(
        __builtins__= safe,
        os= types.SimpleNamespace(path=os.path),
        datetime= datetime,
        platform= platform,
        theLocation= self.location(),
        theWebReader= self,
        theFile= self.theWeb.webFileName,
        thisApplication= sys.argv[0],
        __version__= __version__,
        )
    # Evaluate
    result= str(eval(expression, globals))
except Exception as e:
    self.logger.error( 'Failure to process {!r}: result is {!r}'.format(expression, e) )
    self.errors += 1
    result= "@({!r}: Error {!r}@)".format(expression, e)
self.aChunk.appendText( result, self.tokenizer.lineNumber )

◊ add an expression command to the current chunk (125). Used by: minor commands... (121)

A double command sequence ('@@', when the command is an '@') has the usual meaning of '@' in the input stream. We do this via the appendText() method of the current Chunk. This will append the character on the end of the most recent TextCommand; if this fails, it will create a new, empty TextCommand.

We replace with '@' here and now! This is put this at the end of the previous chunk. And we make sure the next chunk will be appended to this so that it's largely seamless.

double at-sign replacement, append this character to previous TextCommand (126) =

self.aChunk.appendText( self.command, self.tokenizer.lineNumber )

◊ double at-sign replacement, append this character to previous TextCommand (126). Used by: minor commands... (121)

The expect() method examines the next token to see if it is the expected item. '\n' are absorbed. If this is not found, a standard type of error message is raised. This is used by handleCommand().

WebReader handle a command string (127) +=

def expect( self, tokens ):
    try:
        t= next(self.tokenizer)
        while t == '\n':
            t= next(self.tokenizer)
    except StopIteration:
        self.logger.error( "At {!r}: end of input, {!r} not found".format(self.location(),tokens) )
        self.errors += 1
        return
    if t not in tokens:
        self.logger.error( "At {!r}: expected {!r}, found {!r}".format(self.location(),tokens,t) )
        self.errors += 1
        return
    return t

◊ WebReader handle a command string (127). Used by: WebReader class... (114)

The location() provides the file name and line number. This allows error messages as well as tangled or woven output to correctly reference the original input files.

WebReader location in the input stream (128) =

def location( self ):
    return (self.fileName, self.tokenizer.lineNumber+1)

◊ WebReader location in the input stream (128). Used by: WebReader class... (114)

The load() method reads the entire input file as a sequence of tokens, split up by the Tokenizer. Each token that appears to be a command is passed to the handleCommand() method. If the handleCommand() method returns a True result, the command was recognized and placed in the Web. If handleCommand() returns a False result, the command was unknown, and we write a warning but treat it as text.

The load() method is used recursively to handle the @i command. The issue is that it's always loading a single top-level web.

WebReader load the web (129) =

def load( self, web, filename, source=None ):
    self.theWeb= web
    self.fileName= filename

    # Only set the a web filename once using the first file.
    # This should be a setter property of the web.
    if self.theWeb.webFileName is None:
        self.theWeb.webFileName= self.fileName

    if source:
        self._source= source
        self.parse_source()
    else:
        with open( self.fileName, "r" ) as self._source:
            self.parse_source()

def parse_source( self ):
        self.tokenizer= Tokenizer( self._source, self.command )
        self.totalFiles += 1

        self.aChunk= Chunk() # Initial anonymous chunk of text.
        self.aChunk.webAdd( self.theWeb )

        for token in self.tokenizer:
            if len(token) >= 2 and token.startswith(self.command):
                if self.handleCommand( token ):
                    continue
                else:
                    self.logger.warn( 'Unknown @-command in input: {!r}'.format(token) )
                    self.aChunk.appendText( token, self.tokenizer.lineNumber )
            elif token:
                # Accumulate a non-empty block of text in the current chunk.
                self.aChunk.appendText( token, self.tokenizer.lineNumber )

◊ WebReader load the web (129). Used by: WebReader class... (114)

The command character can be changed to permit some flexibility when working with languages that make extensive use of the @ symbol, i.e., PERL. The initialization of the WebReader is based on the selected command character.

WebReader command literals (130) =

# Structural ("major") commands
self.cmdo= self.command+'o'
self.cmdd= self.command+'d'
self.cmdlcurl= self.command+'{'
self.cmdrcurl= self.command+'}'
self.cmdlbrak= self.command+'['
self.cmdrbrak= self.command+']'
self.cmdi= self.command+'i'

# Inline ("minor") commands
self.cmdlangl= self.command+'<'
self.cmdrangl= self.command+'>'
self.cmdpipe= self.command+'|'
self.cmdlexpr= self.command+'('
self.cmdrexpr= self.command+')'
self.cmdcmd= self.command+self.command

# Content "minor" commands
self.cmdf= self.command+'f'
self.cmdm= self.command+'m'
self.cmdu= self.command+'u'

◊ WebReader command literals (130). Used by: WebReader class... (114)

The Tokenizer Class

The WebReader requires a tokenizer. The tokenizer breaks the input text into a stream of tokens. There are two broad classes of tokens:

• @. command tokens, including the structural, inline, and content commands.
• \n. Inside text, these matter. Within structure command tokens, these don't matter. Except after the filename after an @i command, where it ends the command.
• The remaining text.

The tokenizer works by reading the entire file and splitting on @. patterns. The split() method of the Python re module will separate the input and preserve the actual character sequence on which the input was split. This breaks the input into blocks of text separated by the @. characters.

This tokenizer splits the input using (r'@.|\n'). The idea is that we locate commands, newlines and the interstitial text as three classes of tokens. We can then assemble each Command instance from a short sequence of tokens. The core TextCommand and CodeCommand will be a line of text ending with the \n.

The re.split() method will include an empty string when the split pattern occurs at the very beginning or very end of the input. For example:

>>> pat.split( "@{hi mom@}")
['', '@{', 'hi mom', '@}', '']

We can safely filter these via a generator expression.

The tokenizer counts newline characters for us, so that error messages can include a line number. Also, we can tangle comments into the file that include line numbers.

Since the tokenizer is a proper iterator, we can use tokens= iter(Tokenizer(source)) and next(tokens) to step through the sequence of tokens until we raise a StopIteration exception.

Imports (131) +=

import re

◊ Imports (131). Used by: pyweb.py (153)

Tokenizer class - breaks input into tokens (132) =

class Tokenizer:
    def __init__( self, stream, command_char='@' ):
        self.command= command_char
        self.parsePat= re.compile( r'({!s}.|\n)'.format(self.command) )
        self.token_iter= (t for t in self.parsePat.split( stream.read() ) if len(t) != 0)
        self.lineNumber= 0
    def __next__( self ):
        token= next(self.token_iter)
        self.lineNumber += token.count('\n')
        return token
    def __iter__( self ):
        return self

◊ Tokenizer class - breaks input into tokens (132). Used by: Base Class Definitions (1)

The Option Parser Class

For some commands (@d and @o) we have options as well as the chunk name or file name. This roughly parallels the way Tcl or the shell works.

The two examples are

• @o which has an optional -start and -end that are used to provide comment bracketing information. For example:

  @0 -start /* -end */ something.css

  Provides two options in addition to the required filename.

• @d which has an optional -noident or -indent that is used to provide the indentation rules for this chunk. Some chunks are not indented automatically. It's up to the author to get the indentation right. This is used in the case of a Python """ string that would be ruined by indentation.

To handle this, we have a separate lexical scanner and parser for these two commands.

Imports (133) +=

import shlex

◊ Imports (133). Used by: pyweb.py (153)

Here's how we can define an option.

OptionParser(
    OptionDef( "-start", nargs=1, default=None ),
    OptionDef( "-end", nargs=1, default="" ),
    OptionDef( "-indent", nargs=0 ), # A default
    OptionDef( "-noindent", nargs=0 ),
    OptionDef( "argument", nargs='*' ),
    )

The idea is to parallel argparse.add_argument() syntax.

Option Parser class - locates optional values on commands (134) =

class OptionDef:
    def __init__( self, name, **kw ):
        self.name= name
        self.__dict__.update( kw )

◊ Option Parser class - locates optional values on commands (134). Used by: Base Class Definitions (1)

The parser breaks the text into words using shelex rules. It then steps through the words, accumulating the options and the final argument value.

Option Parser class - locates optional values on commands (135) +=

class OptionParser:
    def __init__( self, *arg_defs ):
        self.args= dict( (arg.name,arg) for arg in arg_defs )
        self.trailers= [k for k in self.args.keys() if not k.startswith('-')]
    def parse( self, text ):
        try:
            word_iter= iter(shlex.split(text))
        except ValueError as e:
            raise Error( "Error parsing options in {!r}".format(text) )
        options = dict( s for s in self._group( word_iter ) )
        return options
    def _group( self, word_iter ):
        option, value, final= None, [], []
        for word in word_iter:
            if word == '--':
                if option:
                    yield option, value
                try:
                    final= [next(word_iter)]
                except StopIteration:
                    final= [] # Special case of '--' at the end.
                break
            elif word.startswith('-'):
                if word in self.args:
                    if option:
                        yield option, value
                    option, value = word, []
                else:
                    raise ParseError( "Unknown option {0}".format(word) )
            else:
                if option:
                    if self.args[option].nargs == len(value):
                        yield option, value
                        final= [word]
                        break
                    else:
                        value.append( word )
                else:
                    final= [word]
                    break
        # In principle, we step through the trailers based on nargs counts.
        for word in word_iter:
            final.append( word )
        yield self.trailers[0], " ".join(final)

◊ Option Parser class - locates optional values on commands (135). Used by: Base Class Definitions (1)

In principle, we step through the trailers based on nargs counts. Since we only ever have the one trailer, we skate by.

The loop becomes a bit more complex to capture the positional arguments, in order. First, we have to use an OrderedDict instead of a dict.

Then we'd have a loop something like this. (Untested, incomplete, just hand-waving.)

trailers= self.trailers[:] # Stateful shallow copy
for word in word_iter:
    if len(final) == trailers[-1].nargs: # nargs=='*' vs. nargs=int??
        yield trailers[0], " ".join(final)
        final= 0
        trailers.pop(0)
yield trailers[0], " ".join(final)

import pyweb, os, runpy, sys
pyweb.tangle( "source.w" )
with open("source.log", "w") as target:
    sys.stdout= target
    runpy.run_path( 'source.py' )
    sys.stdout= sys.__stdout__
pyweb.weave( "source.w" )

→Action superclass has common features of all actions (137)
→ActionSequence subclass that holds a sequence of other actions (140)
→WeaveAction subclass initiates the weave action (144)
→TangleAction subclass initiates the tangle action (147)
→LoadAction subclass loads the document web (150)

Action Class

The Action class embodies the basic operations of pyWeb. The intent of this hierarchy is to both provide an easily expanded method of adding new actions, but an easily specified list of actions for a particular run of pyWeb.

The overall process of the application is defined by an instance of Action. This instance may be the WeaveAction instance, the TangleAction instance or a ActionSequence instance.

The instance is constructed during parsing of the input parameters. Then the Action class perform() method is called to actually perform the action. There are three standard Action instances available: an instance that is a macro and does both tangling and weaving, an instance that excludes tangling, and an instance that excludes weaving. These correspond to the command-line options.

anOp= SomeAction( parameters )
anOp.options= argparse.Namespace
anOp.web = Current web
anOp()

The Action is the superclass for all actions. An Action has a number of common attributes.

name:	A name for this action.
options:	The argparse.Namespace object.
web:	The current web that's being processed.

!start:

The time at which the action started.

Action superclass has common features of all actions (137) =

class Action:
    """An action performed by pyWeb."""
    def __init__( self, name ):
        self.name= name
        self.web= None
        self.options= None
        self.start= None
        self.logger= logging.getLogger( self.__class__.__qualname__ )
    def __str__( self ):
        return "{!s} [{!s}]".format( self.name, self.web )
    →Action call method actually does the real work (138)
    →Action final summary of what was done (139)

◊ Action superclass has common features of all actions (137). Used by: Action class hierarchy... (136)

The __call__() method does the real work of the action. For the superclass, it merely logs a message. This is overridden by a subclass.

Action call method actually does the real work (138) =

def __call__( self ):
    self.logger.info( "Starting {!s}".format(self.name) )
    self.start= time.process_time()

◊ Action call method actually does the real work (138). Used by: Action superclass... (137)

The summary() method returns some basic processing statistics for this action.

Action final summary of what was done (139) =

def duration( self ):
    """Return duration of the action."""
    return (self.start and time.process_time()-self.start) or 0
def summary( self ):
    return "{!s} in {:0.2f} sec.".format( self.name, self.duration() )

◊ Action final summary of what was done (139). Used by: Action superclass... (137)

ActionSequence Class

A ActionSequence defines a composite action; it is a sequence of other actions. When the macro is performed, it delegates to the sub-actions.

The instance is created during parsing of input parameters. An instance of this class is one of the three standard actions available; it generally is the default, "do everything" action.

This class overrides the perform() method of the superclass. It also adds an append() method that is used to construct the sequence of actions.

ActionSequence subclass that holds a sequence of other actions (140) =

class ActionSequence( Action ):
    """An action composed of a sequence of other actions."""
    def __init__( self, name, opSequence=None ):
        super().__init__( name )
        if opSequence: self.opSequence= opSequence
        else: self.opSequence= []
    def __str__( self ):
        return "; ".join( [ str(x) for x in self.opSequence ] )
    →ActionSequence call method delegates the sequence of ations (141)
    →ActionSequence append adds a new action to the sequence (142)
    →ActionSequence summary summarizes each step (143)

◊ ActionSequence subclass that holds a sequence of other actions (140). Used by: Action class hierarchy... (136)

Since the macro __call__() method delegates to other Actions, it is possible to short-cut argument processing by using the Python *args construct to accept all arguments and pass them to each sub-action.

ActionSequence call method delegates the sequence of ations (141) =

def __call__( self ):
    for o in self.opSequence:
        o.web= self.web
        o.options= self.options
        o()

◊ ActionSequence call method delegates the sequence of ations (141). Used by: ActionSequence subclass... (140)

Since this class is essentially a wrapper around the built-in sequence type, we delegate sequence related actions directly to the underlying sequence.

ActionSequence append adds a new action to the sequence (142) =

def append( self, anAction ):
    self.opSequence.append( anAction )

◊ ActionSequence append adds a new action to the sequence (142). Used by: ActionSequence subclass... (140)

The summary() method returns some basic processing statistics for each step of this action.

ActionSequence summary summarizes each step (143) =

def summary( self ):
    return ", ".join( [ o.summary() for o in self.opSequence ] )

◊ ActionSequence summary summarizes each step (143). Used by: ActionSequence subclass... (140)

WeaveAction Class

The WeaveAction defines the action of weaving. This action logs a message, and invokes the weave() method of the Web instance. This method also includes the basic decision on which weaver to use. If a Weaver was specified on the command line, this instance is used. Otherwise, the first few characters are examined and a weaver is selected.

This class overrides the __call__() method of the superclass.

If the options include theWeaver, that Weaver instance will be used. Otherwise, the web.language() method function is used to guess what weaver to use.

WeaveAction subclass initiates the weave action (144) =

class WeaveAction( Action ):
    """Weave the final document."""
    def __init__( self ):
        super().__init__( "Weave" )
    def __str__( self ):
        return "{!s} [{!s}, {!s}]".format( self.name, self.web, self.theWeaver )

    →WeaveAction call method to pick the language (145)
    →WeaveAction summary of language choice (146)

◊ WeaveAction subclass initiates the weave action (144). Used by: Action class hierarchy... (136)

The language is picked just prior to weaving. It is either (1) the language specified on the command line, or, (2) if no language was specified, a language is selected based on the first few characters of the input.

Weaving can only raise an exception when there is a reference to a chunk that is never defined.

WeaveAction call method to pick the language (145) =

def __call__( self ):
    super().__call__()
    if not self.options.theWeaver:
        # Examine first few chars of first chunk of web to determine language
        self.options.theWeaver= self.web.language()
        self.logger.info( "Using {0}".format(self.options.theWeaver.__class__.__name__) )
    self.options.theWeaver.reference_style= self.options.reference_style
    try:
        self.web.weave( self.options.theWeaver )
        self.logger.info( "Finished Normally" )
    except Error as e:
        self.logger.error(
            "Problems weaving document from {!s} (weave file is faulty).".format(
            self.web.webFileName) )
        #raise

◊ WeaveAction call method to pick the language (145). Used by: WeaveAction subclass... (144)

The summary() method returns some basic processing statistics for the weave action.

WeaveAction summary of language choice (146) =

def summary( self ):
    if self.options.theWeaver and self.options.theWeaver.linesWritten > 0:
        return "{!s} {:d} lines in {:0.2f} sec.".format( self.name,
        self.options.theWeaver.linesWritten, self.duration() )
    return "did not {!s}".format( self.name, )

◊ WeaveAction summary of language choice (146). Used by: WeaveAction subclass... (144)

TangleAction Class

The TangleAction defines the action of tangling. This operation logs a message, and invokes the weave() method of the Web instance. This method also includes the basic decision on which weaver to use. If a Weaver was specified on the command line, this instance is used. Otherwise, the first few characters are examined and a weaver is selected.

This class overrides the __call__() method of the superclass.

The options must include theTangler, with the Tangler instance to be used.

TangleAction subclass initiates the tangle action (147) =

class TangleAction( Action ):
    """Tangle source files."""
    def __init__( self ):
        super().__init__( "Tangle" )
    →TangleAction call method does tangling of the output files (148)
    →TangleAction summary method provides total lines tangled (149)

◊ TangleAction subclass initiates the tangle action (147). Used by: Action class hierarchy... (136)

Tangling can only raise an exception when a cross reference request (@f, @m or @u) occurs in a program code chunk. Program code chunks are defined with any of @d or @o and use @{ @} brackets.

TangleAction call method does tangling of the output files (148) =

def __call__( self ):
    super().__call__()
    self.options.theTangler.include_line_numbers= self.options.tangler_line_numbers
    try:
        self.web.tangle( self.options.theTangler )
    except Error as e:
        self.logger.error(
            "Problems tangling outputs from {!r} (tangle files are faulty).".format(
            self.web.webFileName) )
        #raise

◊ TangleAction call method does tangling of the output files (148). Used by: TangleAction subclass... (147)

The summary() method returns some basic processing statistics for the tangle action.

TangleAction summary method provides total lines tangled (149) =

def summary( self ):
    if self.options.theTangler and self.options.theTangler.linesWritten > 0:
        return "{!s} {:d} lines in {:0.2f} sec.".format( self.name,
        self.options.theTangler.totalLines, self.duration() )
    return "did not {!r}".format( self.name, )

◊ TangleAction summary method provides total lines tangled (149). Used by: TangleAction subclass... (147)

LoadAction Class

The LoadAction defines the action of loading the web structure. This action uses the application's webReader to actually do the load.

An instance is created during parsing of the input parameters. An instance of this class is part of any of the weave, tangle and "do everything" action.

This class overrides the __call__() method of the superclass.

The options must include webReader, with the WebReader instance to be used.

LoadAction subclass loads the document web (150) =

class LoadAction( Action ):
    """Load the source web."""
    def __init__( self ):
        super().__init__( "Load" )
    def __str__( self ):
        return "Load [{!s}, {!s}]".format( self.webReader, self.web )
    →LoadAction call method loads the input files (151)
    →LoadAction summary provides lines read (152)

◊ LoadAction subclass loads the document web (150). Used by: Action class hierarchy... (136)

Trying to load the web involves two steps, either of which can raise exceptions due to incorrect inputs.

1. The WebReader class load() method can raise exceptions for a number of syntax errors as well as OS errors.
   • Missing closing brackets (@}, @] or @>).
   • Missing opening bracket (@{ or @[) after a chunk name (@d or @o).
   • Extra brackets (@{, @[, @}, @]).
   • Extra @|.
   • The input file does not exist or is not readable.
2. The Web class createUsedBy() method can raise an exception when a chunk reference cannot be resolved to a named chunk.

LoadAction call method loads the input files (151) =

def __call__( self ):
    super().__call__()
    self.webReader= self.options.webReader
    self.webReader.command= self.options.command
    self.webReader.permitList= self.options.permitList
    self.web.webFileName= self.options.webFileName
    error= "Problems with source file {!r}, no output produced.".format(
            self.options.webFileName)
    try:
        self.webReader.load( self.web, self.options.webFileName )
        if self.webReader.errors != 0:
            self.logger.error( error )
            raise Error( "Syntax Errors in the Web" )
        self.web.createUsedBy()
        if self.webReader.errors != 0:
            self.logger.error( error )
            raise Error( "Internal Reference Errors in the Web" )
    except Error as e:
        self.logger.error(error)
        raise # Older design.
    except IOError as e:
        self.logger.error(error)
        raise

◊ LoadAction call method loads the input files (151). Used by: LoadAction subclass... (150)

The summary() method returns some basic processing statistics for the load action.

LoadAction summary provides lines read (152) =

def summary( self ):
    return "{!s} {:d} lines from {:d} files in {:0.2f} sec.".format(
        self.name, self.webReader.totalLines,
        self.webReader.totalFiles, self.duration() )

◊ LoadAction summary provides lines read (152). Used by: LoadAction subclass... (150)

→Overheads (155), →(156), →(157)
→Imports (11), →(47), →(96), →(124), →(131), →(133), →(154), →(158), →(164)
→Base Class Definitions (1)
→Application Class (159), →(160)
→Logging Setup (165), →(166)
→Interface Functions (167)

Python Library Imports

Numerous Python library modules are used by this application.

A few are listed here because they're used widely. Others are listed closer to where they're referenced.

• The os module provide os-specific file and path manipulations; it is used to transform the input file name into the output file name as well as track down file modification times.
• The time module provides a handy current-time string; this is used to by the HTML Weaver to write a closing timestamp on generated HTML files, as well as log messages.
• The datetime module is used to format times, phasing out use of time.
• The types module is used to get at SimpleNamespace for configuration.

Imports (154) +=

import os
import time
import datetime
import types

◊ Imports (154). Used by: pyweb.py (153)

Note that os.path, time, datetime and platform` are provided in the expression context.

Overheads

The shell escape is provided so that the user can define this file as executable, and launch it directly from their shell. The shell reads the first line of a file; when it finds the '#!' shell escape, the remainder of the line is taken as the path to the binary program that should be run. The shell runs this binary, providing the file as standard input.

Overheads (155) =

#!/usr/bin/env python

◊ Overheads (155). Used by: pyweb.py (153)

A Python __doc__ string provides a standard vehicle for documenting the module or the application program. The usual style is to provide a one-sentence summary on the first line. This is followed by more detailed usage information.

Overheads (156) +=

"""pyWeb Literate Programming - tangle and weave tool.

Yet another simple literate programming tool derived from nuweb,
implemented entirely in Python.
This produces any markup for any programming language.

Usage:
    pyweb.py [-dvs] [-c x] [-w format] file.w

Options:
    -v           verbose output (the default)
    -s           silent output
    -d           debugging output
    -c x         change the command character from '@' to x
    -w format    Use the given weaver for the final document.
                 Choices are rst, html, latex and htmlshort.
                 Additionally, a `module.class` name can be used.
    -xw          Exclude weaving
    -xt          Exclude tangling
    -pi          Permit include-command errors
    -rt          Transitive references
    -rs          Simple references (default)
    -n           Include line number comments in the tangled source; requires
                 comment start and stop on the @o commands.

    file.w       The input file, with @o, @d, @i, @[, @{, @|, @<, @f, @m, @u commands.
"""

◊ Overheads (156). Used by: pyweb.py (153)

The keyword cruft is a standard way of placing version control information into a Python module so it is preserved. See PEP (Python Enhancement Proposal) #8 for information on recommended styles.

We also sneak in a "DO NOT EDIT" warning that belongs in all generated application source files.

Overheads (157) +=

__version__ = """2.3.2"""

### DO NOT EDIT THIS FILE!
### It was created by /Users/slott/Documents/Projects/pyWeb-2.3/pyweb/pyweb.py, __version__='2.3.2'.
### From source pyweb.w modified Mon Mar 17 10:13:24 2014.
### In working directory '/Users/slott/Documents/Projects/pyWeb-2.3/pyweb'.

◊ Overheads (157). Used by: pyweb.py (153)