Prodfile

Prodfile is a Python script that defines build rules and dependencies for the PyProd build system. It is used to specify how to build a project, including the source files, build targets, and the rules to generate the targets.

Rule definition

A rule is defined using the @rule decorator, which takes the target file as an argument. The target file is the output of the rule, and the function that follows the decorator is the build logic that generates the target file.

@rule(targets, pattern=None, *, depends=(), uses=())

Defines rule to build target files.

Parameters:

targets – The target file or files to be generated by the rule. Wildcards can be used in filenames, and exactly one % must be included in the filename.
pattern (str | Path | None) – Specify the pattern used to extract the stem of the target filename.
depends (str | Path | list[str | Path]) – Specify the dependencies of the target file. The target file will be rebuilt if any of the dependencies are newer than the target file.
uses (str | Path | list[str | Path]) – Specify the dependencies of the target file. Unlike the depends parameter, the target file will not be rebuilt if any of the dependencies are newer than the target file.

Build function

The function following the @rule decorator is the build function that generates the target file.

The first argument of the build function specifies the target file to be generated.
Subsequent arguments correspond to the filenames listed in the depends parameter. The rule function must accept the target and the same number of arguments as those specified in depends.
uses dependencies are not passed to the build function.
Even if Path objects are specified in the rule, all arguments passed to the builder function will be of type str.

For example, the following code prints file1 ['file2', 'file3'] when the target file file1 is built:

@rule("file1", depends=("file2", "file3"), uses=("file4", "file5"))
def builds(target, *deps):
    print(target, deps)

Pattern

A pattern is a filename template that uses % as a wildcard to match any string. The portion of the target matched by % is called the stem, which is used to substitute % in dependencies. For example, in the following rule, a *.o file is built from the corresponding *.c file sharing the same stem.

@rule("%.o", depends="%.c")
def builds(target, src):
    print(target, src)

If % cannot be used directly in the target, you can use the pattern argument to extract the stem. For example, the following rule builds dir1/file1.o from dir1/file1.c:

@rule("dir1/file1.o", pattern="%.o", depends="%.c")
def build(target, src):
    print(target, src)

Flattening Dependencies

Sequences specified for target, depends, and uses are automatically flattened, so there’s no need to manually concatenate lists.

For example, the following code is equivalent to the previous example:

@rule(["file1"], depends=["file2", ["file3"]], uses=[["file4", ["file5"]]])
def builds(target, *deps):
   print(target, deps)

Using rule as a Standalone Function

The rule decorator can also be used as a standalone function without being tied to a specific function. This makes it convenient for specifying dependencies for multiple targets. For example:

rule(targets=("file1", "file2"), depends="inc1")
rule(targets=("file3", "file4"), uses="inc2")

Checker definition

PyProd provides default checkers for common file types for files and directories. For non-file targets requiring specialized checks, you can define a custom checker to determine whether a build is needed. A checker is defined using the @check decorator, which takes the target file as an argument.

@check(targets)

Defines a checker to get last modified time of the target.

param targets:

The target file or files to be checked. Wildcards can be used.

For example, a checker to retrieve the last modified timestamp of a file on Amazon S3 can be defined as follows:

import re, boto3, botocore
s3 = boto3.client("s3")

# Returns bucket and key from s3 URL
def parse_s3url(url):
    return re.match(r"s3://([^/]+)/(.+)", url).groups()

# Builds s3://bucket/key/file.txt if data.txt is newer
@rule("s3://TESTBUCKET/key/file.txt", depends="data.txt")
def build_s3file(target, src):
    bucket, key = parse_s3url(target)
    s3.upload_file(src, Bucket=bucket, Key=key)

# This checker matches "s3://bucket/key/file.txt"
@check("s3://*")
def check_s3file(target):
    """Checks if an S3 file exists. Returns timestamp if it does."""
    bucket, key = parse_s3url(target)
    try:
       return s3.head_object(Bucket=bucket, Key=key)["LastModified"]
    except botocore.exceptions.ClientError as e:
       if e.response["Error"]["Code"] == "404":
           return
       raise

Task definition

A task is similar to a rule but does not have a target and is always executed when it is depended upon.

@task(*, name=None, uses=(), default=False)

Defines a task to be executed.

Parameters:

name (str) – The name of the task. Defaults to the function name.
uses (str | Path | list[str | Path]) – Specify the dependencies of the target file.
default – If True, this task will be executed when no target is specified in the command-line arguments.

@task(depends=("file1", "file2"))
def my_task(*files):
    print("Task executed", files)

@task can be used without any arguments if no dependencies are specified.

@task
def my_task(*files):
    print("Task executed", files)

Built-in Functions/Variables

In addition to the @rule and @check decorators, PyProd provides several other built-in functions that can be used without importing them. These functions are designed to facilitate various aspects of the build process.

The following built-ins are available:

build(*deps)

Schedule dependencies. The specified deps are built sequentially after the current build completes.

Parameters:: deps – name of dependencies to be built.

Example:

@task
def rebuild():
    build(clean, EXE)

pip(*args)

Install Python packages. It creates a virtual environment if one does not already exist and installs the specified packages.

Parameters:: args – Arguments to pass to the pip install command.

Example:

pip("numpy", "pandas")

run(*args, echo=True, shell=None, stdout=None, cwd=None, text=True, check=True)

Execute a command. This function is a wrapper around subprocess.run() and provides additional functionality for the build system.

Parameters:

args (str | Path| list[str | Path]) – Command and arguments to execute. If first argument is a list, the first element is the command and the rest are arguments. Sequences specified for args are automatically flattened.
echo (bool) – Print the command before executing it (default True).
shell (bool) – Run the command in a shell. If None, the shell is used unless arg is sequence (default None).
stdout – Capture the output of the command (default False).
cwd – Change the current working directory before executing the command.
text – Use text mode for stdout and stderr (default True).
check – Raise an exception if the command returns a non-zero exit code (default True).

Returns:

Returns instance of CompletedProcess.

Examples:

run(["echo", "Hello, World!"]) # list style args
run("echo Hello, World") # Shell style args
run("echo", "Hello,", "World") # Shell style args (automactic concatenation)
run("echo", ["hello", ["world"]]) # Shell style args (automactic flattening)

files = run("ls", stdout=True).stdout # Capture output

capture(*args, echo=True, cwd=None, check=True, text=True, shell=None)

Execute a command and capture the output. This function is a wrapper around run.

Parameters:

args (str | Path | list[str | Path]) – Command and arguments to execute. If first argument is a list, the first element is the command and the rest are arguments. Sequences specified for args are automatically flattened.
echo (bool) – Print the command before executing it (default True).
cwd – Change the current working directory before executing the command.
check – Raise an exception if the command returns a non-zero exit code (default True).
text – Use text mode for stdout and stderr (default True).
shell (bool) – Run the command in a shell. If None, the shell is used unless arg is sequence (default None).

Returns:

Returns the output of the command as a string. Trimmed of trailing newline.

Examples:

msg = capture("echo Hello, World!")

read(filename)

Read the contents of a file.

Parameters:: filename (str | Path) – The file to read.
Returns:: The contents of the file.
Return type:: str

write(filename, txt, append=False)

Write text to a file.

Parameters:

filename (str | Path) – The file to write to.
txt (str) – The text to write.
append (bool) – Append to the file instead of overwriting it (default False).

makedirs(path)

Create a directory along with any necessary parent directories if they do not already exist. This function wraps os.makedirs() with the exists_ok parameter set to True.

Parameters:: path (str | Path) – The directory to create.

glob(path, dir='.')

Glob the given relative pattern in the directory represented by this path. This function is a wrapper around pathlib.Path.glob(). Unlike pathlib.Path.glob(), this function ignores files and directlies that start with a dot. Also, this function returns a list of Path objects.

Parameters:

path (str | Path) – The file pattern to match.
dir (str | Path) – The directory to search in (default .).

Returns:

A list of Path object.

Return type:

list[Path]

Examples:

SRCFILES = glob("**/*.c")

quote(*s)

q(*s)

Quote strings in s. Each s is flattend.

Parameters:: s (str | list) – The string to quote.
Returns:: The list of quoted strings.
Return type:: list[str]

squote(s)

sq(s)

Convert s to string and quote for use as a shell command argument. This function is a wrapper around shlex.quote().

Parameters:: s (str | list) – The string to quote. If s is sequence, it is flattened and joined with space.
Returns:: The quoted string.
Return type:: str

use_git(use)

Enable or disable git support. If enabled, PyProd retrieves the last modified time of the files from the git log.

Parameters:: use – Enable or disable git support.

class Path

A class representing file paths. This function is an alias for pathlib.Path.

Returns:: A Path object.
Return type:: Path

shutil: An alias for the Python Standard Library’s shutil module. This module provides a higher-level interface for file operations than the built-in os module.

env

A dictionary that holds environment variables. You can also access values using dot notation, like env.NAME. Unlike os.environ, env returns an empty string (“”) if a variable is not set.

Examples:

print(env["UNKNOWN_ENV_VAR"]) # prints ""
print(env.PATH) # prints the value of the PATH environment variable
env.VAR = "value" # sets the value of the VAR environment variable

params

A dictionary that holds variables passed from the Command line options. You can also access values using dot notation, like params.NAME. params returns an empty string (“”) if a variable is not set.

Examples:

print(params["UNKNOWN_ENV_VAR"]) # prints ""
print(env.PATH) # prints the value of the PATH environment variable
env.VAR = "value" # sets the value of the VAR environment variable

MAX_TS: A constant representing the maximum timestamp. This value can be is used to force a target to be rebuilt.