The Template-Based Approach

Best practices for designing, coding, and distributing your Python software

Learn Python development best practices from an expert, with detailed coverage of naming and coding conventions

Apply object-oriented principles, design patterns, and advanced syntax tricks

Manage your code with distributed version control

Profile and optimize your code

Practice test-driven development and continuous integration

For more information, please visit:
http://www.PacktPub.com/expert-python-programming/book

The boiler-plate code in acme.sql is composed of a tree of folders that create the namespace of a few files in the root folder. To make all packages follow the same structure, a generic code template can be extracted and provided through a code-generation tool. This approach, called generative programming, is very useful at the organization level. It standardizes the way the code is written and makes developers more productive, as they focus on the code they really need to create. This approach is also a good opportunity to prepare a few things in the package such as complex test fixtures that are common to several packages.

There are numerous generative tools available in the community, but the most used is probably Python Paste (http://pythonpaste.org).

Python Paste

The Python Paste project was partly responsible for the success of frameworks such as Pylons (http://pylonshq.com). Developers are driven by an extensive suite of templates that lets them create applications’ skeletons within minutes.

From the official tutorial, this is a three-liner to create a web application and run it:

$ paster create -t pylons helloworld
$ cd helloworld
$ paster serve --reload development.ini

The Plone and Zope communities followed this philosophy, and now provide Python Paste templates to generated skeletons as well. ZopeSkel (http://pypi.python.org/pypi/ZopeSkel) is one of them.

Python Paste contains several tools, and the template engine we are interested in is PasteScript. It can be installed with
easy_install. It will get all dependencies from the Paste project:

$ easy_install PasteScript
Searching for PasteScript
Reading http://pypi.python.org/simple/PasteScript/
Reading http://pythonpaste.org/script/
Best match: PasteScript 1.6.2
Downloading
. . .
Processing dependencies for PasteScript
Searching for PasteDeploy
...
Searching for Paste>=1.3
...
Finished processing dependencies for PasteScript

The paster command will be available with a few default templates than can be listed with the
–list-templates option of the create command:

$ paster create --list-templates
Available templates:
basic_package: A basic setuptools-enabled package
paste_deploy: A web application deployed through paste.deploy

The basic_package is almost what acme.sql would have needed to build a namespaced package with a
setup.py file. When run, the command line asks a few questions and the corresponding answers will be used to fill the templates:

$ paster create -t basic_package mypackage
Selected and implied templates:
PasteScript#basic_package A basic setuptools-enabled package
...
Enter version (Version (like 0.1)) ['']: 0.1
Enter description ['']: My package
Enter long_description ['']: this is the package
Enter keywords ['']: package is mine
Enter author (Author name) ['']: Tarek
Enter author_email (Author email) ['']: tarek@ziade.org
Enter url (URL of homepage) ['']: http://ziade.org
Enter license_name (License name) ['']: GPL
Enter zip_safe [False]:
Creating template basic_package
...

The resulting structure is a valid, setuptools-compliant, one-level structure:

$ find mypackage
mypackage
mypackage/mypackage
mypackage/mypackage/__init__.py
mypackage/setup.cfg
mypackage/setup.py

Creating Templates

Python Paste, let’s call it the paster, can work with the Cheetah template engine for instance (http://cheetahtemplate.org), and feed it with the user input.

To create a new template for the paster, three elements have to be provided:

A class derived from paste.script.templates.Template
The structure to be created that contains folder and files (Cheetah templates or static files)
A setuptools entry point to paste.paster_create_template, to register the class

Creating the Package Template

Let’s create the template that would have been used for acme.sql.

All templates created here, including the
package are gathered in
pbp.skels that is available for your convenience at PyPI. So if you don’t
want to create your own from scratch, install it:
$ easy_install pbp.skels
This section has step-by-step instructions explaining how pbp.skels
was created.

To create the package template, the first thing to do is to create a structure for this new package:

$ mkdir -p pbp.skels/pbp/skels
$ find pbp.skels
pbp.skels
pbp.skels/pbp
pbp.skels/pbp/skels

Then, an __init__.py file with the following code is created in the
pbp folder. It tells distutils to make it a namespaced package:

try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

Next, create a setup.py file in the root folder (path_to_pbp_package/pbp.skels/_ _init__.py) with the right metadata. The correct code for this is shown here:

from setuptools import setup, find_packages
version = '0.1.0'
classifiers = [
"Programming Language :: Python",
("Topic :: Software Development :: "
"Libraries :: Python Modules")]
setup(name='pbp.skels',
version=version,
description=("PasteScript templates for the Expert "
"Python programming Book."),
classifiers=classifiers,
keywords='paste templates',
author='Tarek Ziade',
author_email='tarek@ziade.org',
url='http://atomisator.ziade.org',
license='GPL',
packages=find_packages(exclude=['ez_setup']),
namespace_packages=['pbp'],
include_package_data=True,
install_requires=['setuptools',
'PasteScript'],
entry_points="""
# -*- Entry points: -*-
[paste.paster_create_template]
pbp_package = pbp.skels.package:Package
""")

The entry point adds a new template that will be available in the paster.

The next step is to write the Package class in the pbp/skels folder, in a module called
package:

from paste.script.templates import var
from paste.script.templates import Template
class Package(Template):
"""Package template"""
_template_dir = 'tmpl/package'
summary = "A namespaced package with a test environment"
use_cheetah = True
vars = [
var('namespace_package', 'Namespace package',
default='pbp'),
var('package', 'The package contained',
default='example'),
var('version', 'Version', default='0.1.0'),
var('description',
'One-line description of the package'),
var('author', 'Author name'),
var('author_email', 'Author email'),
var('keywords', 'Space-separated keywords/tags'),
var('url', 'URL of homepage'),
var('license_name', 'License name', default='GPL')
]
def check_vars(self, vars, command):
if not command.options.no_interactive and \
not hasattr(command, '_deleted_once'):
del vars['package']
command._deleted_once = True
return Template.check_vars(self, vars, command)

This class defines:

The folder containing the template structure(_template_dir)
A summary of the template that will appear in the paster
A flag to indicate if Cheetah is used in the template structure
A list of variables, where each variable is composed of a name, a label, and a default value (if needed), which is used by the paster to ask the user at the prompt to enter his or her values
A check_vars method that makes sure the package variable will be requested at the prompt

The last thing to do is to create the tmpl/package directory content by copying the one created for
acme.sql. All files that contain values to be changed, such as the namespace, have to be suffixed by
_tmpl. The values are replaced by ${variable}, where variable is the name of the variable listed in the
Package class.

The setup.py file (for instance) becomes setup.py_tmpl and contains:

from setuptools import setup, find_packages
import os
version = ${repr($version) or "0.0"}
long_description = open("README.txt").read()
classifiers = [
"Programming Language :: Python",
("Topic :: Software Development :: "
"Libraries :: Python Modules")]
setup(name=${repr($project)},
version=version,
description=${repr($description) or $empty},
long_description=long_description,
classifiers=classifiers,
keywords=${repr($keywords) or $empty},
author=${repr($author) or $empty},
author_email=${repr($author_email) or $empty},
url=${repr($url) or $empty},
license=${repr($license_name) or $empty},
packages=find_packages(exclude=[‘ez_setup’]),
namespace_packages=[${repr($namespace_package)}],
include_package_data=True,
install_requires=[
‘setuptools’,
# -*- Extra requirements: -*-
],
test_suite=’nose.collector’,
test_requires=[‘Nose’],
entry_points=”””
# -*- Entry points: -*-
“””,
)

The repr function will tell Cheetah to add quotes around the string values.

You can use the same technique for all files located in acme.sql to make a template. For instance, the
README.txt file is copied to README.txt_tmpl. Then all references to
acme.sql are replaced by values defined in the Package class in the
vars list.

For instance, getting the full package name is done by:

${namespace_package}.${package}

Last, to use a variable value for a folder name it has to be named with a “+” prefix and suffix. For instance, the namespaced package folder will be called
+namespace_ package+ and the package folder +package+.

The final structure of pbp.skeles, after the acme.sql has been generalized, will look like this:

$ cd pbp.skels
$ find .
setup.py
pbp
pbp/__init__.py
pbp/skels
pbp/skels/__init__.py
pbp/skels/package.py
pbp/skels/tmpl
pbp/skels/tmpl/package
pbp/skels/tmpl/package/README.txt_tmpl
pbp/skels/tmpl/package/setup.py_tmpl
pbp/skels/tmpl/package/+namespace_package+
pbp/skels/tmpl/package/+namespace_package+/__init__.py_tmpl
pbp/skels/tmpl/package/+namespace_package+/+package+
pbp/skels/tmpl/package/+namespace_package+/+package+/__init__.py
---

From there, the package can be symlinked to Python’s site-packages directory with a
develop command, and made available to the paster:

$ python setup.py develop
...
Finished processing dependencies for pbp.skels==0.1.0dev

After the develop command is run, you should find the template listed in paster :

$ paster create --list-templates
Available templates:
basic_package: A basic setuptools-enabled package
pbp_package: A namespaced package with a test environment
paste_deploy: A web application ... paste.deploy
$ paster create -t pbp_package trying.it
Selected and implied templates:
pbp.skels#package A namespaced package with a test environment
Variables:
egg: trying.it
package: tryingit
project: trying.it
Enter namespace_package (Namespace package) ['pbp']: trying
Enter package (The package contained) ['example']: it
...
Creating template package
...

The generated tree will then contain the structure ready to work with right away.

Development Cycle

The development cycle of a package is composed of iterations, where the code is moved forward from an initial state to a new state. This phase lasts mostly for a few weeks and ends with a release. This does not happen in small packages that are very simple to work with, but can be found in all packages that have enough modules to make it worthwhile.

At the end of the iteration, a release is created with the commands we have previously seen. The package moves at this moment from a development state to a releasable state, and the delivered code can be seen as an official release.

Then a new cycle starts with an incremented version for the package.

What Version Numbers Should be Used?

There are no fixed conventions for incrementing a package’s version number, and when developers feel the software has grown a lot, they often jump to a higher number that does not follow the previous series.

Most software usually start with a very small value and uses two or three digits. Sometimes an alphabet letter is appended to it when they are trying to finalize a version. rc suffixes are also used to mark a
release candidate. That is a version in test phase where some fixes might be done:

0.1, 0.2, 0.3
0.1.0, 0.1.1, 0.1.2a, 0.1.2b
0.1, 0.2rc1, 0.2rc2

You should decide of your own convention as long as the versions stay consistent all the way. In companies, there are usually standards followed by all applications; whereas open-source applications have their own conventions.

The only rule that should be applied is to make sure that the number of digits is always the same, and avoid the “–” sign in the version, because it is used as a separator by many tools to extract a version number from a package name.

For instance, these should be avoided:

0.1, 0.1.1-alpha, 0.1.1-b, 0.2
0.1, 0.1-a, 0.1-b

Nightly Builds

If the package is still releasable anytime during the iteration, development releases can be made. Those are also called nightly builds. This continuous releasing process allows developers to get live feedback on their work, and save beta users some work. They don’t need to get the code from a version repository, for instance, and can install the development release like a regular one.

To differentiate a development release from a regular release, the user has to append the dev suffix to the version number. For instance, the
0.1.2 version that is being developed and not yet released, will be known as the
0.1.2dev release.

distutils provide a way to mark this, by adding in a setup.cfg file a section that informs the
build command about the development state:

[egg_info]
tag_build = dev

This will automatically add the dev prefix added to the version:

$ python setup.py bdist_egg
running bdist_egg
running egg_info
...
creating 'dist/iw.selenium-0.1.0dev-py2.4.egg'

Another useful tag can be the revision number when the package is living in Subversion repository. It can be appended with the
tag_svn_revision flag:

[egg_info]
tag_build = dev
tag_svn_revision = true

The revision number will appear in the version as well in that case.

$ python setup.py bdist_egg
running bdist_egg
running egg_info
...
creating 'dist/iw.selenium-0.1.0dev_r38360-py2.4.egg'

The simplest way is to always keep this file in the trunk and remove it right before making a regular release. In other words, a releasing process with Subversion can be:

Make a tag copy of the trunk.
Check out the tag branch.
Remove the setup.cfg (or the egg_info-specific section) in this branch and commit the change.
Build the release from there.
Raise the version number in the trunk.

This looks as follows:

$ svn cp http://example.com/my.package/trunk http://example.com/
my.package/tags/0.1
$ svn co http://example.com/my.package/tags/0.1 0.1
$ cd 0.1
$ svn rm setup.cfg
$ svn ci -m “removing the dev flag”"
$ python setup.py register sdist bdist_egg upload

Summary

In this article we have seen:

The template-based approach to generated package skeletons
How The Paster works and how to create a package skeleton
How to release the package and provide nightly builds.

Share and Enjoy: