meshed.makers#

Makers

This module contains tools to make meshed objects in different ways.

Main entry points:

code_to_dag: turn a function whose body is out = func(args...) lines (or such a function’s source string) into a DAG.
code_to_fnodes: the same parsing, but returning the tuple of FuncNode objects instead of assembling a DAG.
src_to_func_node_factory: the lower-level step yielding FuncNode factories (partials that still lack their func).
mk_fnodes_from_fn_factories: inject functions into those factories to get FuncNode objects.

Let’s start with an example where we have some code representing a user story:

>>> def user_story():
...     wfs = call(src_to_wf, data_src)
...     chks_iter = map(chunker, wfs)
...     chks = chain(chks_iter)
...     fvs = map(featurizer, chks)
...     model_outputs = map(model, fvs)

If the code is compliant (has only function calls and assignments of their result), we can extract FuncNode factories from these lines (uses AST behind the scenes).

>>> from meshed.makers import src_to_func_node_factory
>>> fnodes_factories = list(src_to_func_node_factory(user_story))

Each factory is a curried version of FuncNode, set up to be able to make a DAG equivalent to the user story, once we provide the necessary functions (call, map, and chain).

>>> from functools import partial
>>> assert all(
... isinstance(x, partial) and issubclass(x.func, FuncNode) for x in fnodes_factories
... )

See that the FuncNode factories are all set up with name (id), out (output variable name), bind (names of the variables where the function will source it’s arguments), and func_label (which can be used when displaying the DAG, or as a key to the function to use).

>>> assert [x.keywords for x in fnodes_factories] == [
...  {'name': 'call',
...   'out': 'wfs',
...   'bind': {0: 'src_to_wf', 1: 'data_src'},
...   'func_label': 'call'},
...  {'name': 'map',
...   'out': 'chks_iter',
...   'bind': {0: 'chunker', 1: 'wfs'},
...   'func_label': 'map'},
...  {'name': 'chain',
...   'out': 'chks',
...   'bind': {0: 'chks_iter'},
...   'func_label': 'chain'},
...  {'name': 'map_04',
...   'out': 'fvs',
...   'bind': {0: 'featurizer', 1: 'chks'},
...   'func_label': 'map'},
...  {'name': 'map_05',
...   'out': 'model_outputs',
...   'bind': {0: 'model', 1: 'fvs'},
...   'func_label': 'map'}
... ]

What can we do with that?

Well, provide the functions, so the DAG can actually compute.

You can do it yourself, or get a little help with mk_fnodes_from_fn_factories.

>>> from meshed.dag import DAG
>>> from meshed.makers import mk_fnodes_from_fn_factories
>>> fnodes = list(mk_fnodes_from_fn_factories(fnodes_factories))
>>> dag = DAG(fnodes)
>>> print(dag.synopsis_string())
src_to_wf,data_src -> call -> wfs
chunker,wfs -> map -> chks_iter
chks_iter -> chain -> chks
featurizer,chks -> map_04 -> fvs
model,fvs -> map_05 -> model_outputs

Wait! But we didn’t actually provide the functions we wanted to use! What happened?!? What happened is that mk_fnodes_from_fn_factories just made some for us. It used the convenient meshed.util.mk_place_holder_func which makes a function (that happens to actually compute something and be picklable).

>>> from inspect import signature
>>> str(signature(dag))
'(src_to_wf, data_src, chunker, featurizer, model)'

We can actually call the dag and get something meaningful:

>>> dag(1, 2, 3, 4, 5)
'map(model=5, fvs=map(featurizer=4, chks=chain(chks_iter=map(chunker=3, wfs=call(src_to_wf=1, data_src=2)))))'

If you don’t want mk_fnodes_from_fn_factories to do that (because you are in prod and need to make sure as much as possible is explicitly as expected, you can simply use a different factory_to_func argument. The default one is:

>>> from meshed.makers import dlft_factory_to_func

which you can also reuse to make your own. See below how we provide a name_to_func_map to specify how func_label values should map to actual functions, and set use_place_holder_fallback=False to make sure that we don’t ever fallback on a placeholder function as we did above.

>>> def _call(x, y):
...     # would use operator.methodcaller('__call__') but doesn't have a __name__
...     return x + y
>>> def _map(x, y):
...     return [x, y]
>>> def _chain(iterable):
...     return sum(iterable)
>>>
>>> factory_to_func = partial(
...     dlft_factory_to_func,
...     name_to_func_map={'map': _map, 'chain': _chain, 'call': _call},
...     use_place_holder_fallback=False
... )
>>>
>>> fnodes = list(mk_fnodes_from_fn_factories(fnodes_factories, factory_to_func))
>>> dag = DAG(fnodes)

On the surface, we get the same dag as we had before – at least from the point of view of the dag signature, names, and relationships between these names:

>>> print(dag.synopsis_string())
src_to_wf,data_src -> call -> wfs
chunker,wfs -> map -> chks_iter
chks_iter -> chain -> chks
featurizer,chks -> map_04 -> fvs
model,fvs -> map_05 -> model_outputs
>>> str(signature(dag))
'(src_to_wf, data_src, chunker, featurizer, model)'

But see below that the dag is now using the functions we specified:

>>> # dag(src_to_wf=1, data_src=2, chunker=3, featurizer=4, model=5)
>>> # will trigger this:
>>> # src_to_wf=1, data_src=2 -> call -> wfs == 1 + 2 == 3
>>> # chunker=3 , wfs=3 -> map -> chks_iter == [3, 3]
>>> # chks_iter=6 -> chain -> chks == 3 + 3 == 6
>>> # featurizer=4, chks=6 -> map_04 -> fvs == [4, 6]
>>> # model=5, fvs=[4, 6] -> map_05 -> model_outputs == [5, [4, 6]]
>>> dag(1, 2, 3, 4, 5)
[5, [4, 6]]

Functions

`attr_dict`(obj)	Map every attribute name of `obj` not starting with an underscore to its value.
`code_to_dag`([src, func_src, ...])	Build a `DAG` from Python code whose lines are `out = func(args...)` calls.
`code_to_digraph`(src)	Make a `graphviz.Digraph` of the `DAG` that `code_to_dag(src)` builds.
`code_to_fnodes`([src, func_src, ...])	Parse `out = func(args...)` code into a tuple of `FuncNode` objects.
`dag_to_jdict`(dag, *[, func_to_jdict])	Will produce a json-serializable dictionary from a dag.
`dlft_factory_to_func`(factory[, ...])	Get a function for the given factory, looking its `func_label` up in `name_to_func_map`.
`fnode_to_jdict`(fnode, *[, func_to_jdict])	Serialize a `FuncNode` to a dict of its `name`, `func_label`, `bind` and `out`.
`func_nodes_to_named_funcs`(func_nodes)	Make some components (kwargs) based on the `.out` and `.func` of the `FuncNode` objects.
`is_from_ast_module`(o)	Tell whether the class of `o` reports `_ast` as its module.
`iterize`(func)
`jdict_to_dag`(jdict, *[, jdict_to_func])	Will produce a dag from a json-serializable dictionary.
`jdict_to_fnode`(jdict, *[, jdict_to_func])	Rebuild a `FuncNode` from a dict made by `fnode_to_jdict`.
`lined_dag`(funcs)	Chain `funcs` into a `DAG` where each function's output feeds the first parameter of the next.
`mk_fnodes_from_fn_factories`(fnodes_factories)	Make func nodes from func node factories and a specification of how to make the nodes from these.
`named_funcs_to_func_nodes`(named_funcs)	Make `FuncNode` objects from keyword arguments, using the key as the `.out` of the `FuncNode` and the value as the `.func` of the `FuncNode`.
`node_kwargs_to_func_node_factory`(node_kwargs)	Curry `FuncNode` with `node_kwargs` (`name`, `out`, `bind`, ...), leaving `func` to be supplied.
`parse_assignment`(body[, info])	Split an assignment statement into its `(target, call)` ast nodes.
`parse_assignment_steps`(src)	Parse source code and generate tuples of information about it.
`parse_body`(body, *[, body_index])	Turn one body statement into a `(target, call)` pair of ast nodes, or `None`.
`parse_steps`(src)	Parse source code and generate tuples of information about it.
`parsed_to_node_kwargs`(target_value)	Extract FuncNode kwargs (name, out, and bind) from ast (target,value) pairs
`robust_ast_parse`(src)	Parse `src` with `ast.parse`, retrying with the common leading indent stripped on `IndentationError`.
`signed_itemgetter`(*keys)	Like `operator.itemgetter`, except has a signature, which we needed
`simple_code_to_digraph`(src)	Make a `graphviz.Digraph` of the `DAG` that `code_to_dag(src)` builds.
`src_to_func_node_factory`(src[, exclude_names])
`triples_to_fnodes`(triples)	Converts an iterable of func call triples to an iterable of `FuncNode` objects.

Classes

dlft_factory_to_func_mapping()

Mapping view of dlft_factory_to_func: m[factory] is dlft_factory_to_func(factory).

meshed.makers.attr_dict(obj)[source]#: Map every attribute name of obj not starting with an underscore to its value.

meshed.makers.code_to_digraph(src)[source]#: Make a graphviz.Digraph of the DAG that code_to_dag(src) builds.

meshed.makers.dag_to_jdict(dag, *, func_to_jdict=None)[source]#

Will produce a json-serializable dictionary from a dag.

meshed.makers.dlft_factory_to_func(factory, name_to_func_map=None, use_place_holder_fallback=True)[source]#

Get a function for the given factory, looking its func_label up in name_to_func_map.

If the label is missing from the map, a placeholder function (see meshed.util.mk_place_holder_func) is made unless use_place_holder_fallback is False, in which case KeyError is raised.

class meshed.makers.dlft_factory_to_func_mapping[source]#

Bases: Mapping

Mapping view of dlft_factory_to_func: m[factory] is dlft_factory_to_func(factory).

Only __getitem__ is defined, so a subclass must add __iter__ and __len__ before it can be instantiated.

meshed.makers.extract_tokens(string, pos=0, endpos=9223372036854775807)#: Return a list of all non-overlapping matches of pattern in string.

meshed.makers.fnode_to_jdict(fnode, *, func_to_jdict=None)[source]#

Serialize a FuncNode to a dict of its name, func_label, bind and out.

The function itself is included (under func) only when func_to_jdict is given to serialize it.

meshed.makers.func_nodes_to_named_funcs(func_nodes)[source]#

Make some components (kwargs) based on the .out and .func of the FuncNode objects.

Example use: To get from DAG to Slabs.

Return type:: Mapping[str, Callable]

>>> from meshed import DAG, FuncNode
>>> dag = DAG([
...     FuncNode(lambda x: x + 1, out='a'),
...     FuncNode(lambda a: a + 2, out='b',),
...     FuncNode(lambda a, b: a * b, out='c'),
... ])
>>> dag(x=10)
143
>>> named_funcs = func_nodes_to_named_funcs(dag.func_nodes)
>>> isinstance(named_funcs, dict)
True
>>> list(named_funcs)
['a', 'b', 'c']
>>> callable(named_funcs['a'])
True
>>> assert dag.find_func_node('a').func(3) == named_funcs['a'](3) ==4

The inverse of this function is named_funcs_to_func_nodes.

>>> func_nodes = list(named_funcs_to_func_nodes(named_funcs))
>>> dag2 = DAG(func_nodes)
>>> assert dag2(x=3) == dag(x=3) == 24

meshed.makers.is_from_ast_module(o)[source]#

Tell whether the class of o reports _ast as its module.

Note that on Python 3.12 the ast node classes report ast, not _ast, so this returns False for them.

meshed.makers.jdict_to_dag(jdict, *, jdict_to_func=None)[source]#

Will produce a dag from a json-serializable dictionary.

meshed.makers.jdict_to_fnode(jdict, *, jdict_to_func=None)[source]#

Rebuild a FuncNode from a dict made by fnode_to_jdict.

jdict_to_func is required to turn jdict["func"] back into a callable; without it, NotImplementedError is raised.

meshed.makers.lined_dag(funcs)[source]#

Chain funcs into a DAG where each function’s output feeds the first parameter of the next.

Edges are added with DAG.add_edges, which raises ValueError if a function’s first parameter already carries the name of another function in funcs.

meshed.makers.mk_fnodes_from_fn_factories(fnodes_factories, factory_to_func=<function dlft_factory_to_func>)[source]#

Make func nodes from func node factories and a specification of how to make the nodes from these.

Parameters:

fnodes_factories (Iterable[Callable[..., FuncNode]]) – An iterable of FuncNodeFactory
factory_to_func (Callable[[Callable[..., FuncNode]], Callable]) – A function that will give you a function given a FuncNodeFactory input (where it will draw the information it needs to know what kind of function to make).

Return type:

Iterator[FuncNode]

Returns:

meshed.makers.named_funcs_to_func_nodes(named_funcs)[source]#

Make FuncNode objects from keyword arguments, using the key as the .out of the FuncNode and the value as the .func of the FuncNode.

Example use: To get from Slabs to DAG.

Return type:: Iterable[FuncNode]

>>> from meshed import DAG
>>> func_nodes = list(named_funcs_to_func_nodes(dict(
...     a=lambda x: x + 1,
...     b=lambda a: a + 2,
...     c=lambda a, b: a * b)
... ))
>>> dag = DAG(func_nodes)
>>> dag(x=3)
24

The inverse of this function is func_nodes_to_named_funcs.

>>> named_funcs = func_nodes_to_named_funcs(dag.func_nodes)
>>> dag2 = DAG(named_funcs_to_func_nodes(named_funcs))
>>> assert dag2(x=3) == dag(x=3) == 24

meshed.makers.node_kwargs_to_func_node_factory(node_kwargs)[source]#

Curry FuncNode with node_kwargs (name, out, bind, …), leaving func to be supplied.

Return type:: Callable[[Callable], FuncNode]

meshed.makers.parse_assignment(body, info=None)[source]#

Split an assignment statement into its (target, call) ast nodes.

Raises ValueError if body is not an (annotated) assignment, and AssertionError if it has several targets or its value is not a call. The info argument is ignored (it is recomputed from body).

Return type:: tuple

meshed.makers.parse_assignment_steps(src)#

Parse source code and generate tuples of information about it.

Parameters:: src – The source string or a python object whose code string can be extracted.
Returns:: And generator of “target_values”

>>> from meshed.makers import parse_steps
>>> def foo():
...     x = func1(a, b=2)
...     y = func2(x, c=3)
>>> target_values = list(parse_steps(foo))

Let’s look at the first target_value to see what it contains:

>>> name, call = target_values[0]  # a 2-tuple
>>> assert isinstance(name, ast.Name)  # the first element is a ast Name object
>>> sorted(vars(name))
['col_offset', 'ctx', 'end_col_offset', 'end_lineno', 'id', 'lineno']
>>> name.id
'x'
>>> assert isinstance(call, ast.Call)  # the first element is a ast Call object
>>> sorted(vars(call))
['args', 'col_offset', 'end_col_offset', 'end_lineno', 'func', 'keywords', 'lineno']
>>> call.args[0].id
'a'
>>> call.keywords[0].arg
'b'
>>> call.keywords[0].value.value
2

Basically, these ast objects contain all we need to know about the (parsed) source.

meshed.makers.parse_body(body, *, body_index=None)[source]#

Turn one body statement into a (target, call) pair of ast nodes, or None.

Assignments go through parse_assignment; a bare call gets the dummy target _{body_index}; return statements and string constants (docstrings) give None (skipped); anything else raises ValueError.

meshed.makers.parse_steps(src)[source]#

Parse source code and generate tuples of information about it.

Parameters:: src – The source string or a python object whose code string can be extracted.
Returns:: And generator of “target_values”

>>> from meshed.makers import parse_steps
>>> def foo():
...     x = func1(a, b=2)
...     y = func2(x, c=3)
>>> target_values = list(parse_steps(foo))

Let’s look at the first target_value to see what it contains:

>>> name, call = target_values[0]  # a 2-tuple
>>> assert isinstance(name, ast.Name)  # the first element is a ast Name object
>>> sorted(vars(name))
['col_offset', 'ctx', 'end_col_offset', 'end_lineno', 'id', 'lineno']
>>> name.id
'x'
>>> assert isinstance(call, ast.Call)  # the first element is a ast Call object
>>> sorted(vars(call))
['args', 'col_offset', 'end_col_offset', 'end_lineno', 'func', 'keywords', 'lineno']
>>> call.args[0].id
'a'
>>> call.keywords[0].arg
'b'
>>> call.keywords[0].value.value
2

Basically, these ast objects contain all we need to know about the (parsed) source.

meshed.makers.parsed_to_node_kwargs(target_value)[source]#

Extract FuncNode kwargs (name, out, and bind) from ast (target,value) pairs

Parameters:: target_value – A (target, value) pair
Return type:: Iterator[dict]
Returns:: A {name:..., out:..., bind:...} dict (meant to be used to curry FuncNode

Where can you make make target_values? With the parse_assignment_steps function.

>>> from meshed.makers import parse_assignment_steps
>>> def foo():
...     x = func1(a, b=2)
...     y = func2(x, func1, c=3, d=x)
>>> for target_value in parse_assignment_steps(foo):
...     for d in parsed_to_node_kwargs(target_value):
...         print(d)
{'name': 'func1', 'out': 'x', 'bind': {0: 'a', 'b': 2}}
{'name': 'func2', 'out': 'y', 'bind': {0: 'x', 1: 'func1', 'c': 3, 'd': 'x'}}

meshed.makers.robust_ast_parse(src)[source]#: Parse src with ast.parse, retrying with the common leading indent stripped on IndentationError.

meshed.makers.signed_itemgetter(*keys)[source]#: Like operator.itemgetter, except has a signature, which we needed

meshed.makers.simple_code_to_digraph(src)#: Make a graphviz.Digraph of the DAG that code_to_dag(src) builds.

meshed.makers.src_to_func_node_factory(src, exclude_names=None)[source]#

Parameters:

src – Callable or string of callable.
exclude_names – Names to exclude when making func_nodes

Return type:

Iterator[FuncNode | Callable[..., FuncNode]]

Returns:

meshed.makers.triples_to_fnodes(triples)[source]#

Converts an iterable of func call triples to an iterable of FuncNode objects. (Which in turn can be converted to a DAG.)

Note how the python identifiers are extracted (on the basis of “an unbroken sequence of alphanumerical (and underscore) characters”, ignoring all other characters).

Return type:: Iterable[FuncNode]

>>> from meshed import DAG
>>> dag = DAG(
...     triples_to_fnodes(
...     [
...         ('alpha bravo', 'charlie', 'delta echo'),
...         (' foxtrot  &^$#', 'golf', '  alpha,  echo'),
...     ])
... )
>>> print(dag.synopsis_string())
delta,echo -> charlie -> alpha__bravo
alpha__bravo -> alpha__0 -> alpha
alpha__bravo -> bravo__1 -> bravo
alpha,echo -> golf -> foxtrot