So the after pipeline run hook has access to the catalog as a live object so you should be able to access all of the SQL dataset instances

It's a bit hacky but feels doable

Yeah, but its all about the hacky stuffs

I will make a test and let you Know if works

We are using an old 0.16.x version, so, sadly, im not able to grab the engine

Ah - I would encourage you to upgrade at some point if you can

Is in my current roadmap, but like 6 month in the future hahaha

dynamic parameters on datasets in Kedro https://stackoverflow.com/questions/73430557/dynamic-parameters-on-datasets-in-kedro

How to use generators with kedro? https://stackoverflow.com/questions/73460511/how-to-use-generators-with-kedro

Hi, i am currently trying using kedro for a ML prediction pipeline which has a component of preprocessing and other one of prediction. Our client wants to combine this pipeline with another development that has been done in javascript and then use both of them via an API which i have implemented in FastAPI. Since i am consuming the javascript development via API i want to be able to capture the last output of the kedro pipeline into a memory dataset so i can be able to use its output and then provide a Json answer though the API. I am having problems getting this output, since i always get an empty dictionary. I am using in the catalog a MemoryDataset and i specified the copy mode to assign but it still does not work. My concrete question is does anyone know how can i import the pipeline into a python script and get the output of the pipeline to a variable in python so i can use it for further purposes? This is my script template

from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from pathlib import Path



metadata = bootstrap_project(Path("Mypath"))
session=KedroSession.create(metadata.package_name,Path("Mypath"),env="base")
context=session.load_context()

result=session.run(pipeline_name="__default__")
print(result)

thanks for your help

So we can work to find some solution for this, but could you not use a persistence layer to just pick up the data in the downstrem process?

Yes, well this is the way i am doing it but since the data can grow quite fast i would like to keep it in memory. Our client is more concerned in performance. I am not sure if this is possible to do (easily) but i have been wandering if this is possible. I appreciate your help 😄

So I'm 90% sure the last items in the pipeline should be returned in a dictionary as a result of session.run()

But you need to ensure they are not declared as catalog entries and are implicitly treated as MemoryDataSets

I think @datajoely is correct, it will return the free_outputs, which is defined as pipeline outputs - catalog entries, this is one of the bit that I think we could change, but this is how it is done currently.

When you say that they are not declared in the catalog, i have a problem because when i use the parameter of "env" and i do not register the output, it takes the variable from base environment and as a result i never get the output. I do not know if you could suggest a way around to solve this

So you can explicitly declare MemoryDataSets in your catalog, if you do this in your custom env does it work?

That's what i am doing, declaring it but it still returns an empty dictionary 🥲

If you remove the entry from your base env does it work?

i am using

final_data:
  type: MemoryDataSet
  copy_mode: assign

I will try it and i will let you know

😄 Yeah it did work!!!, i do not know why i never thought on removing it from base environemnt, thanks for your help. 🥳

Amazing! I agree with @noklam we can do better here

Hi, we have upgraded kedro to v0.17.7 but now viz seems to stop working, it always hangs at /usr/local/lib/python3.8/site-packages/hdfs/config.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source

there is no verbose option to check if there are any failures, any inputs how to fix

Can u upgrade kedro-viz?

have done that too, have viz version 5.0.1

Are you using a virtual environment here?

Is there a way to run a DAG so that one or more nodes failing in the middle don't stop the whole pipeline? I'd like it to continue running the nodes that do not depend on the failed nodes, possibly tracking the failed and skipped nodes and throwing an error or log message at the end summarizing what was failed or skipped. I've thought of writing a custom runner for this, but it occurred to me something others may have worked on or been interested in.

On_node_error hook may help here, but may need something more to figure out the dag dependencies