ll.make
provides tools for building projects.
Like make it allows you to specify dependencies between files and actions to be executed when files don't exist or are out of date with respect to one of their sources. But unlike make you can do this in an object oriented way and targets are not only limited to files.
Relevant classes are:
Project
, which is the container for all actions in a project andAction
(and subclasses), which are used to transform input data and read and write files (or other entities like database records).
A simple script that copies a file foo.txt
to bar.txt
reencoding it from "latin-1"
to "utf-8"
in the process looks like
this:
from ll import make class MyProject(make.Project): name = "Acme.MyProject" def create(self): make.Project.create(self) source = self.add(make.FileAction("foo.txt")) temp = source.callattr("decode", "iso-8859-1") temp = temp.callattr("encode", "utf-8") target = self.add(make.FileAction("bar.txt", temp)) self.writecreatedone() p = MyProject() p.create() if __name__ == "__main__": p.build("bar.txt")
def filechanged
(key
):
Get the last modified date (or bigbang
, if the file doesn't exist).
class Level
(object
):
Stores information about the recursive execution of Action
s.
def __init__
(self
, action
, since
, reportable
, reported
=False
):
def __repr__
(self
):
def report
(func
):
Standard decorator for Action.get
methods.
This decorator handles proper reporting of nested action calls. If it isn't
used, only the output of calls to Project.writestep
will be visible
to the user.
class RedefinedTargetWarning
(Warning
):
Warning that will be issued when a target is added to a project and a target with the same key already exists.
def __init__
(self
, key
):
def __str__
(self
):
class UndefinedTargetError
(KeyError
):
Exception that will be raised when a target with the specified key doesn't exist within the project.
def __init__
(self
, key
):
def __str__
(self
):
def getoutputs
(project
, since
, input
):
Recursively iterate through the object input
(if it's a
tuple
, list
or dict
) and return a tuple
containing:
An object (
data
) of the same structure asinput
, where every action object encountered is replaced with the output of that action;A timestamp (
changed
) which the newest timestamp among all the change timestamps of the actions encountered.
If none of the actions has any data newer than since
(i.e. none of
the actions produced any new data) data
will be nodata
.
class Action
(object
):
An Action
is responsible for transforming input data into output
data. It may have no, one or many inputs which themselves may be other actions.
It fetches, combines and transforms the output data of those actions and
returns its own output data.
def get
(self
, project
, since
):
This method (i.e. the implementations in subclasses) is the workhorse of
ll.make
. get
must return the output data of the action if
this data has changed since since
(which is a
datetime.datetime
object in UTC). If the data hasn't changed
since since
the special object nodata
must be returned.
In both cases the action must make sure that the data is internally
consistent, i.e. if the input data is the output data of other actions
self
has to ensure that those other actions update their data too,
independent from the fact whether get
will return new data or not.
Two special values can be passed for since
:
bigbang
This timestamp is older than any timestamp that can appear in real life. Since all data is newer than this,
get
must always return output data.bigcrunch
This timestamp is newer than any timestamp that can appear in real life. Since there can be no data newer than this,
get
can only return output data in this case if ensuring internal consistency resulted in new data.
In all cases get
must set the instance attribute changed
to the timestamp of the last change to the data before returning. In most
cases this if the newest changed
timestamp of the input actions.
def __init__
(self
):
Create a new Action
instance.
def execute
(self
, *args
, **kwargs
):
Execute the action: transform the input data in args
and
kwargs
and return the resulting output data. This method must be
implemented in subclasses.
def getkey
(self
):
Get the nearest key from self
or its inputs. This is used by
ModuleAction
for the filename.
def getargs
(self
):
def getkwargs
(self
):
def call
(self
, *args
, **kwargs
):
Return a CallAction
for calling self
s output with
positional arguments from args
and keyword arguments from
kwargs
.
def getattr
(self
, attrname
):
Return a GetAttrAction
for getting self
s attribute
named attrname
.
def callattr
(self
, attrname
, *args
, **kwargs
):
Return a CallAttrAction
for calling self
s attribute
named attrname
with positional arguments from args
and
keyword arguments from kwargs
.
def __repr__
(self
):
def __iter__
(self
, *args
, **kwargs
):
Return an iterator over the input actions of self
.
def iterallinputs
(self
):
Return an iterator over all input actions of self
(i.e. recursively).
def findpaths
(self
, input
):
Find dependency paths leading from self
to the other action
input
. I.e. if self
depends directly or indirectly on
input
, this generator will produce all paths p
where
p[0] is self
and p[-1] is input
and p[i+1] in p[i]
for all
i
in range(len(p)-1)
.
class ObjectAction
(Action
):
An ObjectAction
returns an object.
def get
(self
, project
, since
):
def __init__
(self
, object
=None
):
def __iter__
(self
):
class TransformAction
(Action
):
A TransformAction
depends on exactly one input action and transforms
the input data into output data.
def __init__
(self
, input
=None
):
def getkey
(self
):
def __iter__
(self
):
def getkwargs
(self
):
class CollectAction
(TransformAction
):
A CollectAction
is a TransformAction
that simply outputs its
input data unmodified, but updates a number of other actions in the process.
def get
(self
, project
, since
):
def __init__
(self
, input
=None
, *otherinputs
):
def addinputs
(self
, *otherinputs
):
Register all actions in otherinputs
as additional actions that have
to be updated before self
is updated.
def __iter__
(self
):
def __repr__
(self
):
class PhonyAction
(Action
):
A PhonyAction
doesn't do anything. It may depend on any number of
additonal input actions which will be updated when this action gets updated.
If there's new data from any of these actions, a PhonyAction
will
return None
(and nodata
otherwise as usual).
def get
(self
, project
, since
):
def __init__
(self
, *inputs
, **kwargs
):
Create a PhonyAction
object. doc
describes the action and
is printed by the method Project.writephonytargets
.
def addinputs
(self
, *inputs
):
Register all actions in inputs
as additional actions that have to
be updated once self
is updated.
def __iter__
(self
):
def __repr__
(self
):
class FileAction
(TransformAction
):
A FileAction
is used for reading and writing files (and other
objects providing the appropriate interface).
def get
(self
, project
, since
):
If a FileAction
object doesn't have an input action it reads the
input file and returns the content if the file has changed since
since
(otherwise nodata
is returned).
If a FileAction
object does have an input action and the output
data from this input action is newer than the file self.key
the data
will be written to the file. Otherwise (i.e. the file is up to date) the
data will be read from the file.
def __init__
(self
, key
, input
=None
, encoding
=None
, errors
=None
):
Create a FileAction
object with key
as the "filename".
key
must be an object that provides a method open
for
opening readable and writable streams to the file. input
is the
data written to the file (or the action producing the data). encoding
is the encoding to be used for reading/writing. If encoding
is
None
binary i/o will be used. errors
is the codec error
handling name for encoding/decoding text.
def getkey
(self
):
def getkwargs
(self
):
def write
(self
, project
, data
):
Write data
to the file and return it.
def read
(self
, project
):
Read the content from the file and return it.
def chmod
(self
, mode
=420
):
Return a ModeAction
that will change the file permissions of
self
to mode
.
def chown
(self
, user
=None
, group
=None
):
Return an OwnerAction
that will change the user and/or group
ownership of self
.
def __repr__
(self
):
class MkDirAction
(TransformAction
):
This action creates the a directory (passing through its input data).
def __init__
(self
, key
, mode
=511
):
Create a MkDirAction
instance. mode
(which defaults to
0o777
) will be used as the permission bit pattern for the new
directory.
def execute
(self
, project
, data
):
Create the directory with the permission bits specified in the constructor.
def __repr__
(self
):
class PipeAction
(TransformAction
):
This action pipes the input through an external shell command.
def __init__
(self
, input
, command
):
Create a PipeAction
instance. command
is the shell command
to be executed (which must read it's input from stdin and write its output
to stdout).
def getkwargs
(self
):
def execute
(self
, project
, data
, command
):
def __repr__
(self
):
class CacheAction
(TransformAction
):
A CacheAction
is a TransformAction
that passes through its
input data, but caches it, so that it can be reused during the same build
round.
def get
(self
, project
, since
):
def __init__
(self
, input
=None
):
class GetAttrAction
(TransformAction
):
This action gets an attribute from its input object.
def __init__
(self
, input
=None
, attrname
=None
):
def __iter__
(self
):
def getkwargs
(self
):
def execute
(self
, project
, data
, attrname
):
class CallAction
(Action
):
This action calls a function or any other callable object with a number of arguments. Both positional and keyword arguments are supported and the function and the arguments can be static objects or actions.
def __init__
(self
, func
, *args
, **kwargs
):
def __iter__
(self
):
def getargs
(self
):
def getkwargs
(self
):
def execute
(self
, project
, func
, *args
, **kwargs
):
class CallAttrAction
(Action
):
This action calls an attribute of an object with a number of arguments. Both positional and keyword arguments are supported and the object, the attribute name and the arguments can be static objects or actions.
def __init__
(self
, obj
, attrname
, *args
, **kwargs
):
def __iter__
(self
):
def getargs
(self
):
def getkwargs
(self
):
def execute
(self
, project
, obj
, attrname
, *args
, **kwargs
):
class CommandAction
(TransformAction
):
This action executes a system command (via os.system
) and passes
through the input data.
def __init__
(self
, command
, input
=None
):
Create a new CommandAction
object. command
is the command
that will executed when execute
is called.
def execute
(self
, project
, data
):
def __repr__
(self
):
class ModeAction
(TransformAction
):
ModeAction
changes file permissions and passes through the input data.
def __init__
(self
, input
=None
, mode
=420
):
Create an ModeAction
object. mode
(which defaults to
0644
) will be use as the permission bit pattern.
def __iter__
(self
):
def getkwargs
(self
):
def execute
(self
, project
, data
, mode
):
Change the permission bits of the file self.getkey()
.
class OwnerAction
(TransformAction
):
OwnerAction
changes the user and/or group ownership of a file and
passes through the input data.
def __init__
(self
, input
=None
, user
=None
, group
=None
):
Create a new OwnerAction
object. user
can either be a
numerical user id or a user name or None
. If it is None
no user ownership will be changed. The same applies to group
.
def __iter__
(self
):
def getkwargs
(self
):
def execute
(self
, project
, data
, user
, group
):
Change the ownership of the file self.getkey()
.
class ModuleAction
(TransformAction
):
This action will turn the input string into a Python module.
def get
(self
, project
, since
):
def __init__
(self
, input
=None
):
Create an ModuleAction
.
This object must have an input action (which might be a FileAction
that creates the source file).
def addinputs
(self
, *inputs
):
Register all actions in inputs
as modules used by this module.
These actions must be ModuleAction
objects too.
Normally it isn't neccessary to call the method directly. Instead fetch the required module inside your module like this:
from ll import make mymodule = make.currentproject.get("mymodule.py")
This will record your module as depending on mymodule
, so if
mymodule
changes, your module will be reloaded too. For this to
work you need to have an ModuleAction
added to the project with
the key "mymodule.py"
.
def __iter__
(self
):
def execute
(self
, project
, data
):
def __repr__
(self
):
class FOPAction
(TransformAction
):
This action transforms an XML string (containing XSL-FO) into PDF. For it
to work Apache FOP is required. The command line is hardcoded but it's
simple to overwrite the class attribute command
in a subclass.
def execute
(self
, project
, data
):
class AlwaysAction
(Action
):
This action always returns None
as new data.
def get
(self
, project
, since
):
def __init__
(self
):
def __iter__
(self
):
class NeverAction
(Action
):
This action never returns new data.
def get
(self
, project
, since
):
def __iter__
(self
):
class Project
(dict
):
A Project
collects all Action
objects from a project. It
is responsible for initiating the build process and for generating a report
about the progress of the build process.
def __init__
(self
):
def __repr__
(self
):
property showaction:
This property specifies which actions should be reported during the build process. On setting, the value can be:
None
or"none"
No actions will be reported;
"file"
Only
FileAction
s will be reported;"phony"
Only
PhonyAction
s will be reported;"filephony"
Only
FileAction
s andPhonyAction
s will be reported;- a class or tuple of classes
Only actions that are instances of those classes will be reported.
def __get__
(self
):
def __set__
(self
, value
):
property showstep:
This property specifies for which actions tranformation steps should be
reported during the build process. For allowed values on setting see
showaction
.
def __get__
(self
):
def __set__
(self
, value
):
property shownote:
This property specifies which for which actions tranformation notes
(which are similar to step, but not that important, e.g. when an
information that is already there gets reused) be reported during the
build process. For allowed values on setting see showaction
.
def __get__
(self
):
def __set__
(self
, value
):
property showregistration:
This property specifies for which actions registration (i.e. call to the
add
should be reported. For allowed values on setting see
showaction
.
def __get__
(self
):
def __set__
(self
, value
):
def _getenvbool
(self
, name
, default
):
def strtimedelta
(self
, delta
):
Return a nicely formatted and colored string for the
datetime.timedelta
value delta
. delta
may also be None
in with case "0"
will be returned.
def strdatetime
(self
, dt
):
Return a nicely formatted and colored string for the
datetime.datetime
value dt
.
def strcounter
(self
, counter
):
Return a nicely formatted and colored string for the counter value
counter
.
def strerror
(self
, text
):
Return a nicely formatted and colored string for the error text
text
.
def strkey
(self
, key
):
Return a nicely formatted and colored string for the action key
key
.
def straction
(self
, action
):
Return a nicely formatted and colored string for the action
action
.
def strdata
(self
, data
):
def __setitem__
(self
, key
, target
):
Add the action target
to self
as a target and register it
under the key key
.
def add
(self
, target
, key
=None
):
Add the action target
as a target to self
. If key
is not None
, target
will be registered under this key,
otherwise it will be registered under its own key (i.e. target.key
).
def _candidates
(self
, key
):
Return candidates for alternative forms of key
. This is a
generator, so when the first suitable candidate is found, the rest of the
candidates won't have to be created at all.
def __getitem__
(self
, key
):
Return the target with the key key
. If an key can't be found, it
will be wrapped in a ll.url.URL
object and retried. If
key
still can't be found a UndefinedTargetError
will be
raised.
def has_key
(self
, key
):
Return whether the target with the key key
exists in the project.
def __contains__
(self
, key
):
Return whether the target with the key key
exists in the project.
def create
(self
):
Create all dependencies for the project. Overwrite in subclasses.
This method should only be called once, otherwise you'll get lots of
RedefinedTargetWarning
s. But you can call clear
to remove all targets before calling create
. You can also
use the method recreate
for that.
def recreate
(self
):
Calls clear
and create
to recreate all project
dependencies.
def argparser
(self
):
Return an argparse
parser for parsing the command line arguments.
This can be overwritten in subclasses to add more arguments.
def parseargs
(self
, args
=None
):
Use the parser returned by argparser
to parse the argument
sequence args
, modify self
accordingly and return
the result of the parsers parse_args
call.
def _get
(self
, target
, since
):
target
must be an action registered in self
(or the id of
one). For this target the Action.get
will be called with
since
as the argument.
def get
(self
, target
):
Get up-to-date output data from the target target
(which must be
an action registered with self
(or the id of one). During the call
the global variable currentproject
will be set to self
.
def build
(self
, *targets
):
Rebuild all targets in targets
. Items in targets
must be
actions registered with self
(or their ids).
def buildwithargs
(self
, args
=None
):
For calling make scripts from the command line. args
defaults to
sys.argv
. Any positional arguments in the command line will be treated
as target ids. If there are no positional arguments, a list of all
registered PhonyAction
objects will be output.
def write
(self
, *texts
):
All screen output is done through this method. This makes it possible to redirect the output (e.g. to logfiles) in subclasses.
def writeln
(self
, *texts
):
All screen output is done through this method. This makes it possible to redirect the output (e.g. to logfiles) in subclasses.
def writeerror
(self
, *texts
):
Output an error.
def notifystart
(self
):
def notifyfinish
(self
, duration
, success
):
def warn
(self
, warning
, stacklevel
):
Issue a warning through the Python warnings framework
def writestacklevel
(self
, level
, *texts
):
Output texts
indented level
levels.
def writestack
(self
, *texts
):
Output texts
indented properly for the current nesting of
action execution.
def _writependinglevels
(self
):
def writestep
(self
, action
, *texts
):
Output texts
as the description of the data transformation
done by the action arction
.
def writenote
(self
, action
, *texts
):
Output texts
as the note for the data transformation done by
the action action
.
def writecreatedone
(self
):
Can be called at the end of an overwritten create
to report
the number of registered targets.
def writephonytargets
(self
):
Show a list of all PhonyAction
objects in the project and
their documentation.
def findpaths
(self
, target
, source
):
Find dependency paths leading from the action target
to the action
source
.