How Mush works¶
Note
This documentation explains how Mush works using fairly abstract examples. If you’d prefer more “real world” examples please see the Example Usage documentation.
Constructing runners¶
Mush works by assembling a number of callables into a Runner
:
from mush import Runner
def func1():
print('func1')
def func2():
print('func2')
runner = Runner(func1, func2)
Once assembled, a runner can be called any number of times. Each time it is called, it will call each of its callables in turn:
>>> runner()
func1
func2
More callables can be added to a runner:
def func3():
print('func3')
runner.add(func3)
If you want to add several callables in one go, you can use the
runner’s extend()
method:
def func4():
print('func4')
def func5():
print('func5')
runner.extend(func4, func5)
Now, when called, the runner will call all five functions:
>>> runner()
func1
func2
func3
func4
func5
Runners can also be added together to create a new runner:
runner1 = Runner(func1)
runner2 = Runner(func2)
runner3 = runner1 + runner2
This addition does not modify the existing runners, but does give the result you’d expect:
>>> runner1()
func1
>>> runner2()
func2
>>> runner3()
func1
func2
This can also be done by passing runners in when creating a new runner or calling the extend method on a runner, for example:
runner1 = Runner(func1)
runner2 = Runner(func2)
runner4_1 = Runner(runner1, runner2)
runner4_2 = Runner()
runner4_2.extend(runner1, runner2)
In both cases, the results are as you would expect:
>>> runner4_1()
func1
func2
>>> runner4_2()
func1
func2
Finally, runners can be cloned, providing a way to encapsulate commonly used base runners that can then be extended for each specific use case:
runner5 = runner3.clone()
runner5.add(func4)
The existing runner is not modified, while the new runner behaves as expected:
>>> runner3()
func1
func2
>>> runner5()
func1
func2
func4
Configuring Resources¶
Where Mush becomes useful is when the callables in a runner either produce or require objects of a certain type. Given the right configuration, Mush will wire these together enabling you to write easily testable and reusable callables that encapsulate specific pieces of functionality. This configuration is done either imperatively, declaratively or using a combination of the two styles as described in the sections below.
For the examples, we’ll assume we have three types of resources:
class Apple:
def __str__(self):
return 'an apple'
__repr__ = __str__
class Orange:
def __str__(self):
return 'an orange'
__repr__ = __str__
class Juice:
def __str__(self):
return 'a refreshing fruit beverage'
__repr__ = __str__
Imperative configuration¶
Imperative configuration requires no decorators and is great when working with callables that come from another package or the standard library:
def apple_tree():
print('I made an apple')
return Apple()
def magician(fruit):
print('I turned {0} into an orange'.format(fruit))
return Orange()
def juicer(fruit1, fruit2):
print('I made juice out of {0} and {1}'.format(fruit1, fruit2))
The requirements are passed to the add()
method of a runner which can express requirements for both arguments
and keyword parameters:
runner = Runner()
runner.add(apple_tree)
runner.add(magician, Apple)
runner.add(juicer, fruit1=Apple, fruit2=Orange)
Calling this runner will now manage the resources, collecting them and passing them in as configured:
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
Declarative configuration¶
This is done using the requires()
decorator to mark the
callables with their requirements, which can specify the types
required for either arguments or keyword parameters:
from mush import requires
def apple_tree():
print('I made an apple')
return Apple()
@requires(Apple)
def magician(fruit):
print('I turned {0} into an orange'.format(fruit))
return Orange()
@requires(fruit1=Apple, fruit2=Orange)
def juicer(fruit1, fruit2):
print('I made juice out of {0} and {1}'.format(fruit1, fruit2))
return Juice()
These can now be combined into a runner and executed. The runner will extract the requirements stored by the decorator and will use them to map the parameters as appropriate:
>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange
Hybrid configuration¶
The two styles of configuration are entirely interchangeable, with
declarative requirements being inspected whenever a callable is added
to a runner, and imperative requirements being taken whenever they are
passed via the add()
method:
@requires(Juice)
def packager(juice):
print('I put {0} in a bottle'.format(juice))
def orange_tree():
print('I made an orange')
return Orange()
trees = Runner(apple_tree, orange_tree)
runner = trees.clone()
runner.extend(juicer, packager)
This runner now ends up with bottled juice:
>>> runner()
I made an apple
I made an orange
I made juice out of an apple and an orange
I put a refreshing fruit beverage in a bottle
It’s useful to note that imperative configuration will be used in preference to declarative configuration where both are present:
runner = trees.clone()
runner.add(juicer, Orange, Apple)
This runner will give us juice made in a different order:
>>> runner()
I made an apple
I made an orange
I made juice out of an orange and an apple
Resource usage periods¶
It can be important for a callable to have either first access or last access to a particular resource. For this reason, configuration can specify one of three periods during which a callable needs to be called with a particular resource; the default is normal and a decorator can be used to indicate either first or last. Within these periods, callables are called in the order they are added to the runner.
As an example, consider a ring and some things that can be done to it:
class Ring:
def __str__(self):
return 'a ring'
def forge():
return Ring()
def polish(ring):
print('polishing {0}'.format(ring))
def more_polish(ring):
print('polishing {0} again'.format(ring))
def engrave(ring):
print('engraving {0}'.format(ring))
def package(ring):
print('packaging {0}'.format(ring))
These can now be added to a runner with configuration expressing the correct periods:
from mush import Runner, first, last
runner = Runner(forge)
runner.add(package, last(Ring))
runner.add(polish, first(Ring))
runner.add(more_polish, first(Ring))
runner.add(engrave, Ring)
Even though the callables were added out order, they will be executed correctly:
>>> runner()
polishing a ring
polishing a ring again
engraving a ring
packaging a ring
The configuration of periods works identically with both the imperative and declarative forms.
Waiting for a resource¶
Sometimes, a callable needs to wait for some other callable to do its work but does not need or cannot accept objects of the type returned by that callable. For example, and miss-using the fruit types from above:
def func1():
return Apple()
def func2(apple):
print('func2 got {0}'.format(apple))
return Orange()
def func3(apple):
print('func3 got {0}'.format(apple))
def func4(orange):
print('func4 processed {0}'.format(orange))
If we want func3()
only to get called once func4()
has
processed the Orange
but, for reasons of abstraction,
we want to add the callables in the order defined above, the simplest
runner will not give us what we want:
>>> runner = Runner(func1)
>>> runner.add(func2, Apple)
>>> runner.add(func3, Apple)
>>> runner.add(func4, Orange)
>>> runner()
func2 got an apple
func3 got an apple
func4 processed an orange
The problem is that the runner hasn’t been told that func3()
has a dependency on Orange
. This can be done using the
after()
type wrapper to specify that func2()
requires an
Orange
to exist, and for any other callables added to the
runner that need an Orange
to have been called first, but
that it must not be passed that orange:
from mush import Runner, after
runner = Runner(func1)
runner.add(func2, Apple)
runner.add(func3, after(Orange), Apple)
runner.add(func4, Orange)
Now, even though we’ve added the callables in the order we want, we get the order of calling that we need:
>>> runner()
func2 got an apple
func4 processed an orange
func3 got an apple
Special return types¶
There are certain types that can be returned from a callable that have special meaning. While these are provided in case of their specific need, you should think twice when you find yourself wanting to use them as it is often a sign that your code can be better structured differently.
For the examples below, we’ll use the fruit classes from above.
Returning multiple resources¶
A callable can return multiple resources by returning either a list or a tuple:
def orchard():
return Apple(), Orange()
This can be used to provide both types of fruit:
>>> runner = Runner(orchard, juicer)
>>> runner()
I made juice out of an apple and an orange
Overriding the type of a resource¶
Sometimes you may need to force a returned resource to be of a particular type. When this is the case, a callable can return a dictionary mapping the forced type to the required type:
class Pear:
def __str__(self):
return 'a pear'
def desperation():
print('oh well, a pear will have to do')
return {Apple: Pear()}
We can now make juice even though we don’t have apples:
>>> runner = Runner(orange_tree, desperation, juicer)
>>> runner()
I made an orange
oh well, a pear will have to do
I made juice out of a pear and an orange
If you have no control over a callable that returns an object of the ‘wrong’ type, you have two options. You can either decorate it:
from mush import returns
@returns(Apple)
def make_pears():
print('I made a pear')
return Pear()
Now you can use the pear maker to get juice:
>>> runner = Runner(orange_tree, make_pears, juicer)
>>> runner()
I made an orange
I made a pear
I made juice out of a pear and an orange
If you can’t even decorate the callable, then you can also imperatively indicate that the return type should be overridden:
from mush import returns
def someone_elses_pears():
print('I made a pear')
return Pear()
Now you can use the pear maker to get juice:
>>> runner = Runner(orange_tree, juicer)
>>> runner.add_returning(someone_elses_pears, returns=Apple)
>>> runner()
I made an orange
I made a pear
I made juice out of a pear and an orange
Returning marker types¶
In some circumstances, you may need to ensure that some callables are used only after another callable has done its work, even though that work does not return a resource. This can be achieved using a marker type as follows:
from mush import nothing, marker
@returns(marker('SetupComplete'))
def setup():
print('setting things up')
@requires(after(marker('SetupComplete')))
def body():
print('doing stuff')
Note that the body()
callable does not take any arguments or
parameters; because after()
is used, it is not passed and is used
purely to affect the order of calling:
>>> runner = Runner(body, setup)
>>> runner()
setting things up
doing stuff
Using parts of a resource¶
When pieces of functionality use settings provided either by command
line arguments or from configuration files, it’s often cleaner to
structure that code to recieve the specific setting value rather than
the setting’s container. It’s certainly easier to test. Mush can take
care of the needed wiring when configured to do so using the
attr
and item
helpers:
from mush import Runner, attr, item, requires
class Config(dict): pass
class Args(object):
fruit = 'apple'
tree = dict(fruit='pear')
def parse_args():
return Args()
def read_config():
return Config(fruit='orange')
@requires(attr(Args, 'fruit'),
item(Config, 'fruit'),
item(attr(Args, 'tree'), 'fruit'))
def pick(fruit1, fruit2, fruit3):
print('I picked {0}, {1} and {2}'.format(fruit1, fruit2, fruit3))
picked = []
for fruit in fruit1, fruit2:
if fruit=='apple':
picked.append(Apple())
elif fruit=='orange':
picked.append(Orange())
else:
raise TypeError('You have made a poor fruit choice')
return picked
While the pick()
function remains usable and testable on its
own:
>>> pick('apple', 'orange', 'pear')
I picked apple, orange and pear
[an apple, an orange]
It can also be added to a runner with the other necessary functions and Mush will do the hard work:
>>> runner = Runner(parse_args, read_config, pick, juicer)
>>> runner()
I picked apple, orange and pear
I made juice out of an apple and an orange
Context manager resources¶
A frequent requirement when writing scripts is to make sure that when unexpected things happen they are logged, transactions are aborted, and other necessary cleanup is done. Mush supports this pattern by allowing context managers to be added as callables:
from mush import Runner, requires
class Transactions(object):
def __enter__(self):
print('starting transaction')
def __exit__(self, type, obj, tb):
if type:
print(obj)
print('aborting transaction')
else:
print('committing transaction')
return True
def a_func():
print('doing my thing')
def good_func():
print('I have done my thing')
def bad_func():
raise Exception("I don't want to do my thing")
The context manager is wrapped around all callables that are called after it:
>>> runner = Runner(Transactions, a_func, good_func)
>>> runner()
starting transaction
doing my thing
I have done my thing
committing transaction
This gives it a chance to clear up when things go wrong:
>>> runner = Runner(Transactions, a_func, bad_func)
>>> runner()
starting transaction
doing my thing
I don't want to do my thing
aborting transaction
Debugging¶
Mush makes some heuristic decisions about the order in which to call objects added to a runner. If your expectations of the call order don’t match that used by Mush, it can be confusing to figure out where the difference comes from.
For this reason, when constructing a Runner
you can pass an
optional debug parameter which can be either a boolean True
or a
file-like object. If passed, debug information will be generated
whenever an object is added to the runner.
If True
, this information will be written to
stderr
. If a file-like object is passed instead, the
information will be written to that object.
As an example, consider this code:
from mush import Runner, requires
class T1(object): pass
class T2(object): pass
class T3(object): pass
def makes_t1():
return T1()
@requires(T1)
def makes_t2(obj):
return T2()
@requires(T2)
def makes_t3(obj):
return T3()
@requires(T3, T1)
def user(obj1, obj2):
m.user(type(obj1), type(obj2))
The debug information that would be written looks like this:
>>> import sys
>>> runner = Runner(makes_t1, makes_t2, makes_t3, user, debug=sys.stdout)
Added <function makes_t1 ...> to 'normal' period for <... 'NoneType'> with Requirements()
Current call order:
For <... 'NoneType'>:
normal: <function makes_t1 ...> requires Requirements()
Added <function makes_t2 ...> to 'normal' period for <class 'T1'> with Requirements(T1)
Current call order:
For <... 'NoneType'>:
normal: <function makes_t1 ...> requires Requirements()
For <class 'T1'>:
normal: <function makes_t2 ...> requires Requirements(T1)
Added <function makes_t3 ...> to 'normal' period for <class 'T2'> with Requirements(T2)
Current call order:
For <... 'NoneType'>:
normal: <function makes_t1 ...> requires Requirements()
For <class 'T1'>:
normal: <function makes_t2 ...> requires Requirements(T1)
For <class 'T2'>:
normal: <function makes_t3 ...> requires Requirements(T2)
Added <function user ...> to 'normal' period for <class 'T3'> with Requirements(T3, T1)
Current call order:
For <... 'NoneType'>:
normal: <function makes_t1 ...> requires Requirements()
For <class 'T1'>:
normal: <function makes_t2 ...> requires Requirements(T1)
For <class 'T2'>:
normal: <function makes_t3 ...> requires Requirements(T2)
For <class 'T3'>:
normal: <function user ...> requires Requirements(T3, T1)