pyffi.spells
— High level file operations¶
Note
This module is based on wz’s NifTester module, although nothing of wz’s original code is left in this module.
A toaster, implemented by subclasses of the abstract
Toaster
class, walks over all files in a folder, and applies
one or more transformations on each file. Such transformations are
called spells, and are implemented by subclasses of the
abstract Spell
class.
A spell can also run independently of a toaster and be
applied on a branch directly. The recommended way of doing this is via
the Spell.recurse()
method.
Supported spells¶
For format specific spells, refer to the corresponding module.
Some spells are applicable on every file format, and those are documented here.
-
class
pyffi.spells.
SpellApplyPatch
(toaster=None, data=None, stream=None)¶ Bases:
pyffi.spells.Spell
A spell for applying a patch on files.
-
datainspect
()¶ There is no need to read the whole file, so we apply the patch already at inspection stage, and stop the spell process by returning
False
.Returns: False
Return type: bool
-
Adding new spells¶
To create new spells, derive your custom spells from the Spell
class, and include them in the Toaster.SPELLS
attribute of your
toaster.
-
class
pyffi.spells.
Spell
(toaster=None, data=None, stream=None)¶ Bases:
object
Spell base class. A spell takes a data file and then does something useful with it. The main entry point for spells is
recurse()
, so if you are writing new spells, start with reading the documentation withrecurse()
.-
READONLY
= True¶ A
bool
which determines whether the spell is read only or not. Default value isTrue
. Override this class attribute, and set toFalse
, when subclassing a spell that must write files back to the disk.
-
SPELLNAME
= None¶ A
str
describing how to refer to the spell from the command line. Override this class attribute when subclassing.
-
__init__
(toaster=None, data=None, stream=None)¶ Initialize the spell data.
Parameters:
-
_branchinspect
(branch)¶ Check if spell should be cast on this branch or not, based on exclude and include options passed on the command line. You should not need to override this function: if you need additional checks on whether a branch must be parsed or not, override the
branchinspect()
method.Parameters: branch ( GlobalNode
) – The branch to check.Returns: True
if the branch must be processed,False
otherwise.Return type: bool
-
_datainspect
()¶ This is called after
pyffi.object_models.FileFormat.Data.inspect()
has been called, and beforepyffi.object_models.FileFormat.Data.read()
is called.Returns: True
if the file must be processed,False
otherwise.Return type: bool
-
branchentry
(branch)¶ Cast the spell on the given branch. First called with branch equal to
data
’s children, then the grandchildren, and so on. The default implementation simply returnsTrue
.Typically, you will override this function to perform an operation on a particular block type and/or to stop recursion at particular block types.
Parameters: branch ( GlobalNode
) – The branch to cast the spell on.Returns: True
if the children must be processed,False
otherwise.Return type: bool
-
branchexit
(branch)¶ Cast a spell on the given branch, after all its children, grandchildren, have been processed, if
branchentry()
returnedTrue
on the given branch.Typically, you will override this function to perform a particular operation on a block type, but you rely on the fact that the children must have been processed first.
Parameters: branch ( GlobalNode
) – The branch to cast the spell on.
-
branchinspect
(branch)¶ Like
_branchinspect()
, but for customization: can be overridden to perform an extra inspection (the default implementation always returnsTrue
).Parameters: branch ( GlobalNode
) – The branch to check.Returns: True
if the branch must be processed,False
otherwise.Return type: bool
-
dataentry
()¶ Called before all blocks are recursed. The default implementation simply returns
True
. You can access the data viadata
, and unlike in thedatainspect()
method, the full file has been processed at this stage.Typically, you will override this function to perform a global operation on the file data.
Returns: True
if the children must be processed,False
otherwise.Return type: bool
-
dataexit
()¶ Called after all blocks have been processed, if
dataentry()
returnedTrue
.Typically, you will override this function to perform a final spell operation, such as writing back the file in a special way, or making a summary log.
-
datainspect
()¶ This is called after
pyffi.object_models.FileFormat.Data.inspect()
has been called, and beforepyffi.object_models.FileFormat.Data.read()
is called. Override this function for customization.Returns: True
if the file must be processed,False
otherwise.Return type: bool
-
recurse
(branch=None)¶ Helper function which calls
_branchinspect()
andbranchinspect()
on the branch, if both successful thenbranchentry()
on the branch, and if this is succesful it callsrecurse()
on the branch’s children, and once all children are done, it callsbranchexit()
.Note that
_branchinspect()
andbranchinspect()
are not called upon first entry of this function, that is, when called withdata
as branch argument. Usedatainspect()
to stop recursion into this branch.Do not override this function.
Parameters: branch ( GlobalNode
) – The branch to start the recursion from, orNone
to recurse the whole tree.
-
stream
= None¶ The current
file
being processed.
-
classmethod
toastentry
(toaster)¶ Called just before the toaster starts processing all files. If it returns
False
, then the spell is not used. The default implementation simply returnsTrue
.For example, if the spell only acts on a particular block type, but that block type is excluded, then you can use this function to flag that this spell can be skipped. You can also use this function to initialize statistics data to be aggregated from files, to initialize a log file, and so.
Parameters: toaster ( Toaster
) – The toaster this spell is called from.Returns: True
if the spell applies,False
otherwise.Return type: bool
-
Grouping spells together¶
It is also possible to create composite spells, that is, spells that simply execute other spells. The following functions and classes can be used for this purpose.
-
pyffi.spells.
SpellGroupParallel
(*args)¶ Class factory for grouping spells in parallel.
-
pyffi.spells.
SpellGroupSeries
(*args)¶ Class factory for grouping spells in series.
-
class
pyffi.spells.
SpellGroupBase
(toaster=None, data=None, stream=None)¶ Bases:
pyffi.spells.Spell
Base class for grouping spells. This implements all the spell grouping functions that fall outside of the actual recursing (
__init__()
,toastentry()
,_datainspect()
,datainspect()
, andtoastexit()
).-
ACTIVESPELLCLASSES
= []¶ List of active spells of this group (not instantiated). This list is automatically built when
toastentry()
is called.
-
datainspect
()¶ Inspect every spell with L{Spell.datainspect} and keep those spells that must be cast.
-
spells
= []¶ List of active spell instances.
-
classmethod
toastentry
(toaster)¶
-
classmethod
toastexit
(toaster)¶
-
-
class
pyffi.spells.
SpellGroupParallelBase
(toaster=None, data=None, stream=None)¶ Bases:
pyffi.spells.SpellGroupBase
Base class for running spells in parallel (that is, with only a single recursion in the tree).
-
branchentry
(branch)¶ Run all spells.
-
branchexit
(branch)¶
-
branchinspect
(branch)¶ Inspect spells with
Spell.branchinspect()
(not all checks are executed, only keeps going until a spell inspection returnsTrue
).
-
changed
¶
-
dataentry
()¶ Look into every spell with
Spell.dataentry()
.
-
dataexit
()¶ Look into every spell with
Spell.dataexit()
.
-
Creating toaster scripts¶
To create a new toaster script, derive your toaster from the Toaster
class, and set the Toaster.FILEFORMAT
attribute of your toaster to
the file format class of the files it can toast.
-
class
pyffi.spells.
Toaster
(spellclass=None, options=None, spellnames=None, logger=None)¶ Bases:
object
Toaster base class. Toasters run spells on large quantities of files. They load each file and pass the data structure to any number of spells.
-
ALIASDICT
= {}¶ Dictionary with aliases for spells.
-
DEFAULT_OPTIONS
= {'raisetesterror': False, 'verbose': 1, 'pause': False, 'exclude': [], 'include': [], 'examples': False, 'spells': False, 'interactive': True, 'helpspell': False, 'dryrun': False, 'prefix': '', 'suffix': '', 'arg': '', 'createpatch': False, 'applypatch': False, 'diffcmd': '', 'patchcmd': '', 'series': False, 'skip': [], 'only': [], 'jobs': 32, 'refresh': 32, 'sourcedir': '', 'destdir': '', 'archives': False, 'resume': False, 'gccollect': False, 'inifile': ''}¶ List of spell classes of the particular
Toaster
instance.
-
EXAMPLES
= ''¶ Some examples which describe typical use of the toaster.
-
FILEFORMAT
¶ The file format class (a subclass of
FileFormat
).alias of
FileFormat
-
cli
()¶ Command line interface: initializes spell classes and options from the command line, and run the
toast()
method.
-
get_toast_head_root_ext
(filename)¶ Get the name of where the input file filename would be written to by the toaster: head, root, and extension.
Parameters: filename ( str
) – The name of the hypothetical file to be toasted.Returns: The head, root, and extension of the destination, or (None, None, None)
if--dry-run
is specified.Return type: tuple
of threestr
s
-
get_toast_stream
(filename, test_exists=False)¶ Calls
get_toast_head_root_ext(filename)()
to determine the name of the toast file, and return a stream for writing accordingly.Then return a stream where result can be written to, or in case test_exists is True, test if result would overwrite a file. More specifically, if test_exists is True, then no streams are created, and True is returned if the file already exists, and False is returned otherwise.
-
indent
= 0¶ An
int
which describes the current indentation level for messages.
-
inspect_filename
(filename)¶ Returns whether to toast a filename or not, based on skip_regexs and only_regexs.
-
is_admissible_branch_class
(branchtype)¶ Helper function which checks whether a given branch type should have spells cast on it or not, based in exclude and include options.
>>> from pyffi.formats.nif import NifFormat >>> class MyToaster(Toaster): ... FILEFORMAT = NifFormat >>> toaster = MyToaster() # no include or exclude: all admissible >>> toaster.is_admissible_branch_class(NifFormat.NiProperty) True >>> toaster.is_admissible_branch_class(NifFormat.NiNode) True >>> toaster = MyToaster(options={"include": ["NiProperty", "NiNode"], "exclude": ["NiMaterialProperty", "NiLODNode"]}) >>> toaster.is_admissible_branch_class(NifFormat.NiProperty) True >>> toaster.is_admissible_branch_class(NifFormat.NiNode) True >>> toaster.is_admissible_branch_class(NifFormat.NiAVObject) False >>> toaster.is_admissible_branch_class(NifFormat.NiLODNode) False >>> toaster.is_admissible_branch_class(NifFormat.NiSwitchNode) True >>> toaster.is_admissible_branch_class(NifFormat.NiMaterialProperty) False >>> toaster.is_admissible_branch_class(NifFormat.NiAlphaProperty) True
-
logger
= <Logger pyffi.toaster (NOTSET)>¶ A
logging.Logger
for toaster log messages.
-
msg
(message)¶ Write log message with
logger.info()
, taking into accountindent
.Parameters: message ( str
) – The message to write.
-
msgblockend
(message=None)¶ Acts like
msg()
, but also decreasesindent
before writing the message, but if the message argument isNone
, then no message is printed.
-
options
= {}¶ The options of the toaster, as
dict
.
-
static
parse_inifile
(option, opt, value, parser, toaster=None)¶ Initializes spell classes and options from an ini file.
-
spellnames
= []¶ A list of the names of the spells.
-
toast
(top)¶ Walk over all files in a directory tree and cast spells on every file.
Parameters: top (str) – The directory or file to toast.
-
toast_archives
(top)¶ Toast all files in all archives.
-
top
= ''¶ Name of the top folder to toast.
-
write
(stream, data)¶ Writes the data to data and raises an exception if the write fails, but restores file if fails on overwrite.
-
writepatch
(stream, data)¶ Creates a binary patch for the updated file.
-