#beginners-need-help
Title
# beginners-need-help
m
mjmare
02/15/2022, 11:15 AMHow does one go about when one has a bunch of dataset that need the same treatment? Currently I use a template in catalog.yml to create the input and output Datasets, like so:
Copy code
{% for table in openac_tables %}
{{ table }}:
layer: primary
type: pandas.ParquetDataSet
filepath: data/03_primary/{{table}}.parquet
save_args:
from_pandas:
preserve_index: False
{% endfor %}
{% for table in openac_tables %}
profile_{{ table }}:
layer: qa
type: ac_pipelines.datasets.ProfilingDataSet
filepath: data/08_reporting/profiles/{{table}}.html
{% endfor %} Copy code
def create_pipeline(**kwargs):
from kedro.config import ConfigLoader
conf_paths = ["conf/base", "conf/local"]
conf_loader = ConfigLoader(conf_paths)
table_names = conf_loader.get('*globals.yml')['openac_tables']
return Pipeline([
node(func=lambda x: x,
inputs=tn,
outputs=f'profile_{tn}',
name=f'profile_{tn}',
)
for tn in table_names
]) Copy code
from kedro.framework.session import get_current_session
session = get_current_session()
context = session.load_context()
table_names = context.config_loader.get('*globals.yml')['openac_tables']d
datajoely
02/15/2022, 12:29 PMI'll start by saying the Kedro team aren't fans of dynamic pipelines because it can become super hard to debug
A while ago we started sourcing ideas on best practice here
https://github.com/kedro-org/kedro/discussions/859#discussioncomment-1205270
where the two points here are relevant:
> - Avoid dynamic DAG creation in Kedro unless you really have to
> - If you have to, ensure that the DAG is structurally immutable and only differs in terms of dataset flow.
I would also warn against loading your own context like @User suggests, you can get it working but it becomes a pain parallelise and can interferes with some of key parts of Kedro's run lifecycle behind the scenes
If you really want access to the catalog - the right way to do so is to use lifecycle hooks
Specifically to this situation - the
before_pipeline_runm
mjmare
02/15/2022, 12:45 PM@User If creating dynamic pipelines is not the Kedro Way, how would you tackle my usecase: I have a bunch of DataSets that all need the same treatment (say, generate a pandas-profiling report). Do you really manually create all the output DataSets manually? And if you need another treatment, you do it again? That would create tons of Datasets.
i
Isaac89
02/15/2022, 12:45 PMThanks for correcting ! I didn't know I can interfere with Kedro's run lifecycle in this way !
d
datajoely
02/15/2022, 12:48 PM@User so I have two suggestions:
- I would create a pandas profiling modular pipeline which you can instantiate and reuse any number of times https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/03_modular_pipelines.html#modular-pipelines
- If you need to persist outputs or use them downstream you can actually replace the
MemoryDataSetafter_pipeline_createdA bit similar to this API example here: https://discord.com/channels/778216384475693066/931129686124298291/932711145234645023
m
mjmare
02/15/2022, 1:04 PM@User I had had a look at modular pipelines,but thought that a pipeline of 1 one looked a bit silly, plus parameters are static in Kedro, right? So I cannot generate a series of pipelines (of 1 node each) by looping over my list of DataSets with different parameter values for each.
Re API example: useful!
There is no way to tag DataSets is there? Would be nice to be able select a subset of of the Datasets in the catalog, and then create additional datasets based on that set.
d
datajoely
02/15/2022, 1:05 PM1 node pipelines are absolutely supported
you can override parameters with the modular pipeline syntax
it has a
paramterslet me check with tags
so tags are an attribute of the nodes - you can in theory mutate them in a hook but it's not prettty
possibly pipeline
namespacem
mjmare
02/15/2022, 1:08 PMDataset tagging might be useful, no?
d
datajoely
02/15/2022, 1:10 PMmaybe - but its sort of designed to be immutable
with 1 node pipelines then namespace may have similar effect
m
mjmare
02/15/2022, 1:11 PMI think I see what you mean. Thx
d
datajoely
02/15/2022, 1:11 PMThis sample project may have some useful examples
m
mjmare
02/15/2022, 1:12 PMThe parameters argument refers to the names of parameters. So I cannot change the values, unless I setup all differently named parameters?
d
datajoely
02/15/2022, 1:12 PMso you can mutate the actual values in the
catalogand then refer those in the
parametersm
mjmare
02/15/2022, 1:15 PMI do not follow. In the catalog?
d
datajoely
02/15/2022, 1:15 PMIn the hook you can do
catalog.add({‘params:something’:object}m
mjmare
02/15/2022, 1:18 PMAh I see. You mutate the catalog in the hook before each pipeline (of 1 node). Correct?
The modular spaceflights project looks very useful. Thanks for that and your tips!
d
datajoely
02/15/2022, 1:39 PMYes I think that's how you would do it
if you're able to share your final implementation that would be great to see
I'm keen to document some ideas for future people hitting this problem
a
antony.milne
02/15/2022, 1:41 PMFYI https://github.com/kedro-org/kedro/issues/750 has quite a thorough discussion of this and some more ideas 🙂
(not the original question in that GH issue, but a couple of posts down)
d
DarthGreedius
02/15/2022, 9:24 PMHi. The title of this thread seems relevant to what I'm looking for.
Which isn't exactly what's been discussed up until now...
But I guess, this would be the place to ask.
I have two pipelines I would like to connect in such a way as the first runs completely before the second
there is no pre-existing input/output relationship I can use to do it, as they are now
d
datajoely
02/15/2022, 9:27 PMThe easiest way to achieve that is to make a dataset dependency between pipeline A and B
d
DarthGreedius
02/15/2022, 9:28 PMyes, i gathered as much from the docs
d
datajoely
02/15/2022, 9:28 PMSo you can just pass a
TrueThe other approach is to get creative with you CLI commands
kedro run —pipeline a && kedro run —pipeline bd
DarthGreedius
02/15/2022, 9:29 PMyou mean, just not do it within the kedro/python code at all
d
datajoely
02/15/2022, 9:29 PM&& will run one completely before the second
d
DarthGreedius
02/15/2022, 9:29 PMyeah... that seems like the wise way
d
datajoely
02/15/2022, 9:29 PMI don’t think the object pass through approach is wrong
d
DarthGreedius
02/15/2022, 9:30 PMI'm case it would require heavy modification to both pipelines
because they are both extremely parallel
which is what led me down this train of thought
*in my case
d
datajoely
02/15/2022, 9:31 PMYou should opt for whatever is the easiest for you to manage you mental model and maintain
d
DarthGreedius
02/15/2022, 9:31 PMwise council
thank you
it's funny how often "just don't do it" is the best way to do something
lol
d
datajoely
02/15/2022, 9:32 PMHahah
I like the phrase: “write code for other people to read, especially if that person is future you”
Good luck!
d
DarthGreedius
02/16/2022, 6:01 PMHello, again...
I'm trying to solve the issue I mentioned yesterday by adding the dataset dependency explicitly in the pipelines... so far I've got...
In the catalog.yml:
requirements_met:
type: kedro.io.MemoryDataSet
=====
In nodes.py:
def gather_reqs(req1 = None, req2 = None) -> bool:
if pivot_qry_exist is None:
return False
if primary_validations is None:
return False
return True
=====
In pipeline.py
return Pipeline(
[
node(
gather_reqs,
inputs = [req1, req2],
outputs = "requirements_met",
name = "check_requirements"
),
node (
step,
inputs = ["dataset", "requirements_met"]
outputs = "out"
),
node (
other_step,
inputs = ["other_dataset", "requirements_met"]
outputs = "other_out"
)
]
)
I've set things up so that the requirements are not really there (req1, req) ... but the nodes still run
Any idea what I'm missing?
d
datajoely
02/16/2022, 6:30 PMcan you tell me what
stepother_stepAm I right in saying you only want one of them to run?
d
DarthGreedius
02/16/2022, 6:32 PMHi, they do some feature eng on the dataframe the receive as input
and I need them all to run
but I would like them to run only if requirements are met
d
datajoely
02/16/2022, 6:33 PMokay and why aren't they running?
do you get an error?
d
DarthGreedius
02/16/2022, 6:33 PMthey are running
and the requirements aren't met
as in: req1 and req2 are datasets in the catalog and they do not exist and there is no node in this pipeline to create them
d
datajoely
02/16/2022, 6:35 PMoh I think I understand what you are trying to do
d
DarthGreedius
02/16/2022, 6:35 PMi had my other pipeline write out req1, req2... if it finishes ok.
d
datajoely
02/16/2022, 6:35 PMmaybe the right way to do this is to define a custom DataSet that returns an empty
DFnonethat would be easier to handle in the node right?
d
DarthGreedius
02/16/2022, 6:37 PMhmmm... while I was waiting I tried something: I changed the definition of requirements_met in the catalog to an actual file (TextDataSet)
d
datajoely
02/16/2022, 6:38 PMI just saw this
Copy code
yaml
requirements_met:
type: kedro.io.MemoryDataSetyou don't need to explicitly declare it
d
DarthGreedius
02/16/2022, 6:38 PMso now none of req1, req2 or requirements_met exist... the pipeline tried to run all the nodes but then finally failed because requirements_met doesn't exist.
I tried it without declaring it first
d
datajoely
02/16/2022, 6:39 PMwhat error did you get?
d
DarthGreedius
02/16/2022, 6:39 PMthe pipeline ran and generated its results normally, completely ignored my attempt at a stopcheck node
hehehe
d
datajoely
02/16/2022, 6:43 PMmaybe put a
breakpoint()step_xd
DarthGreedius
02/16/2022, 6:44 PMok, thx.. I'll keep hammering at it
I think I got it... I had typo.
I mistyped the dataset in the check_requirements node...
d
datajoely
02/16/2022, 7:06 PMThat's my kind of error!
d
DarthGreedius
02/16/2022, 7:07 PMif only all things were that simple... indeed.
2 Views