> so how would I get the return of

before_node_run

into a variable. I mostly just use hooks to get the side-effects.

so you could map the

inputs

dict to a new value

but this is more of a philosophical point - we like to think of nodes as pure python functions that have no knowledge of IO operations or execution logic

so if you can do it outside of a node logic it feels more Kedrific

so, the output of

before_node_run

, can be used as an input?

you can mutate the

inputs

provided to the node

not quite sure how I would mutate the inputs from

before_node_run

At this point - maybe we should say the env variable is the simplest solution!

I've made a note to document this as it comes up a lot

one last idea, could you not get the hook to point to a memory_dataset and then read that (in this case the flag for env)

Yeah that's what I was getting to!

nicely put

I ended up doing something slightly amusing. Just combined your ideas used the default project hook to setup the KEDRO_ENV programmatically via python haha.

I like it!

at first I was very confused

maybe we should make that default behaviour 🤔

yea I think that could work quite well.

Hi @User! I was testing this idea but it seams that Hook instances are different when the before_pipeline_run and before_node_run are executed, so the attribute is reset. Is it possible that the Hook class is instantiated twice ? I can see the same attribute in after_catalog_created, before_pipeline_run, but it is None in the berfore_node_run.

hmmm, that's weird. The class is only instantiated once (in

settings.py

) but possibly there's something else going on that's obliterating the attribute. Let me try it out myself and see if it works

Can confirm that it works ok for me. I think you must have some additional logic going on somewhere that is overwriting the attribute somehow?

Do you have multiple hooks?

ok I will look into it. thanks for confirming that is working in your case.

yes

@User If I run it with the Sequential runner it is ok, but with the --parallel flag I have this problem, Is the parallel runner loading the project multiple times ?

On the develop branch, the

get_current_session()

method is removed, how do I get the current session when running?

What do you want from the session? The answer this question will likely be lifecycle hooks

Hi! I'm running the same pipeline on different inputs and scheduling these runs with slurm. Sometimes I'm getting the same runid for different runs, maybe because they are started at the same time. Any suggestion on how to deal with this? Can one change the run id at runtime?

In the data validation hooks example (https://kedro.readthedocs.io/en/stable/07_extend_kedro/02_hooks.html#add-data-validation), the

DATASET_EXPECTATION_MAPPING

is defined in the class itself:

class DataValidationHooks:

    # Map expectation to dataset
    DATASET_EXPECTATION_MAPPING = {
        "companies": "raw_companies_dataset_expectation",
        "preprocessed_companies": "preprocessed_companies_dataset_expectation",
    }
    ...

Is it possible to define this in the parameters yml?

before_node_run

and

after_node_run

doesn't seem to pass in the

context

.

@User you should be able to retrieve the parameters from the

catalog

object

How does one go about when one has a bunch of dataset that need the same treatment? Currently I use a template in catalog.yml to create the input and output Datasets, like so:

{% for table in openac_tables %}
{{ table }}:
  layer: primary
  type: pandas.ParquetDataSet
  filepath: data/03_primary/{{table}}.parquet
  save_args:
    from_pandas:
      preserve_index: False
{% endfor %}

{% for table in openac_tables %}
profile_{{ table }}:
  layer: qa
  type: ac_pipelines.datasets.ProfilingDataSet
  filepath: data/08_reporting/profiles/{{table}}.html
{% endfor %}

and then generate nodes in the pipeline:

def create_pipeline(**kwargs):
    from kedro.config import ConfigLoader

    conf_paths = ["conf/base", "conf/local"]
    conf_loader = ConfigLoader(conf_paths)
    table_names = conf_loader.get('*globals.yml')['openac_tables']

    return Pipeline([
        node(func=lambda x: x,
             inputs=tn,
             outputs=f'profile_{tn}',
             name=f'profile_{tn}',
             )
        for tn in table_names
    ])

It works. But it feels hacky. It could be improved if I could get the default config_loader from somewhere. I had some success with:

from kedro.framework.session import get_current_session

    session = get_current_session()
    context = session.load_context()
    table_names = context.config_loader.get('*globals.yml')['openac_tables']

but that confuses Kedro viz (Error: There is no active Kedro session.) More substantial improvement would be if the Pipeline/Node could be dynamically parametrized (at runtime). Don't know if that is the right term. I want to feed a variable number of Datasets to a pipeline )or node). I'm probably doing something wrong, so suggestions are welcome.