is this some newbie mistake I'm making?

oh alright then

I just removed the kwargs from the constructor and it worked

it works!

tysm

Amazing!

Hello all! When attempting to run

kedro viz

I get the attached error. The kerdro versions I'm running are below and I did

pip install kedro[all]

however, still experience the error. kedro==0.18.2 kedro-telemetry==0.2.1 kedro-viz==4.7.1

Disregard, it was a typo needed to capitalize the "S" in DataSet😅

You got it right!

is there a way to make a variable declared in a ProjectContext accessible during runtime? I noticed that in the documentation all of the parameters provided (https://kedro.readthedocs.io/en/latest/kedro.framework.context.KedroContext.html#kedro.framework.context.KedroContext) are read only, but I need to pass a reference to my session object to code in nodes in my pipeline...

Hi @brewski hooks are the right way to access the session, context etc during the lifecycle of the run. That being said we expect nodes to be pure python functions so, by design, it's not really possible to do this. My general rule is 'if you're trying to create the context directly, you've gone too far' What are you trying to do exactly? We can help come up with a more kedrific solution

I'm just trying to use certain operations like dask's client.gather on a series of returned futures, so that I can get my results back before moving into the next node -- In my previous posts I figured out how to have a dask session that dictates where the computation is done (which is held in a ProjectContext), and in trying to port some of my previous experience with dask to kedro, I have to maintain some reference to the cluster and client to do these function calls. (Under the hood I think they require an active session to be able to have a place to return the results to).

@deepyaman is our dask person and should be able to answer this.

Hes US based but will online in a couple of hours

I just figured out how to get it out of

context.class_variable_I_just_initialized

in kedro ipython -- I just want to access this same variable within a node

it's not so much a dask question anymore imo

Well the node isn't passed that unless you define a custom runner

Hooks are your best bet I think

Okay- any advice from which end I should start? The documentation looks very intimidating

We have a trivial example of how to do an executive timer there, perhaps that's a good way to get your head around things

Behind the scenes we use a lib called pluggy designed by the pytest folks to handle their extension ecosystem

would I use hooks to inject a node with the project context variable I'm using?

So you can affect node execution, but you can either side of the operation

Hey fellow Kedroids, has anyone had success using an external secrets manager with kedro. I think this is the last thing I need to update to get my team from 0.17.7 to 0.18. Currently I am getting errors in my tests telling me that credential keys are missing, and warnings showing me that I do not have any credential files. We use aws secrets manager with a custom hook that loads the credentials and sets them during register_catalog.

Hi! Is there a way in Kedro to do parameter unpacking (

**dict

) when defining a node, or maybe there's an alternative way of achieving what I want: I want to merge two datasets using

pd.merge

and pass it a set of parameters I define in config, but don't want to pass specific parameters à la

["params:merge.on", "params:merge.how"]

, as I want to keep the node as general as possible, so I can pass different sets of parameters, such as

left_on

or

right_index

. I'd want to just unpack

[**"params:merge"]

values as kwargs. I could write a node which does that before passing to

pd.merge

, but I would like to avoid that if possible as it is something I would like to do for multiple different functions, not just

pd.merge

.

@inigohrey for that you can simply pass params:merge, which is a dictionary. In your node:

node = (some_function, ["some_dataset", "params:merge"])

In your actual fuction:

def merge(some_data, config):
  pd.merge(some_data, **config)
  ...

Yeah, was wondering if there was a way to do that without having to define an intermediate fn to do the unpacking as it isn't just

pd.merge

that I want to interact with in this way. Could maybe write a general wrapper which does this

Hey, not sure if it has been asked already. What are the best practices for loading in and saving data from and within catalogs and how should data flow between raw->intermediate->primary, etc. Also how can I get access to the context catalog. I am using Kedro v0.18.2.

I wrote this s while back, hope it helps!