Hello :) I was wondering whether someone could help me with my current kedro question: I have a function which takes two args: a, b And I would like to wrap it as a node. The pb I am facing is that a is meant to be a dataset from my catalog but b is meant to be a list of literal strings. In the pipeline.py file, when defining the node, I don't manage to define the inputs kwarg

I found a fix for now: I add a node before the node in question that outputs the hard-coded strings I need as a MemoryDataset... I'll see if I can use the params: syntax instead. Not sure whether this allows be to pass sequences of str literals

what about b is defined in the parameter yaml ?

So, I want b to be a tuple of strings, or a list of strings. How can I define this in the parameter yaml? The examples I find online only look like nested dicts

You're going to want something like: yam_str: - yam_1 - yam_2 - yam_n You can then pass params:yam_str to your pipeline node and the arg will pass a list like ["yam_1", "yam_2", "yam_n"]

Thank you so much! It's exactly what I needed. It worked

awesome!

@bgereke thanks for helping out! You have been upgraded to status 🙂

Hi! Is there a fundamental reason why versioning is not supported for PartitionedDataSet or is it more a matter of some added complexity in implementing it? We're considering using kedro versioning feature, and it's a slight annoyance that one of the datasets we use regularly need to be managed differently.

I think we've always been wary of the combinatorial complexity. One approach people have opted for is to use S3 or Delta table versioning provided by the filesystem rather than Kedro.

We could also add it, but it's also not something users have been demanding very loudly.

I used to have the same problem, and I love the delta versioning (but it may be quite difficult for kedro to handle this) since having the entire directory versioned is quite inefficient and I gave up.

Just to be clear, by "use S3", do you mean using S3 bucket with versioning enabled? I'd love to hear what people use and like in practice. I know of DVC of course, but weren't 100% convinced we needed to go there. @noklam You "used to have the same problem", meaning you use Delta table versioning?

No, for my case it's wasn't an input but some intermediate file so I end up didn't version it at all

Hi, is there any "dry-run"-like functionality within Kedro? Sometimes we want to test run a pipeline without overwriting node outputs. Maybe with a hook we could modify the dataset types. The functionality I'm talking about would also be doable by renaming the inputs and outputs as it would force all the datasets to be MemoryDataSets, but it isn't very clean

Potentially also we could define a debug environment and redefine each dataset definition that we don't want to overwrite as a memorydataset, as we would still need the original input datasets defined. But I'm wondering if anybody has had a similar need and has found an efficient way to work this way.

https://github.com/kedro-org/kedro/issues/1160#issuecomment-1023966910 This is similar to what we want just the inverse, as OP wanted to save additional datasets when debugging and not when running normally. The difference for what I am looking for is that the base catalog.yml is always loaded, so we would have to explicitly, dataset by dataset, redefine them as MemoryDataSets right?

There is a dry runner example here https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/run_a_pipeline.html#custom-runners

Thanks @datajoely ! I think that lists all the nodes without actually running them, but maybe we could adapt it to replace certain tagged nodes' input/output. Though I don't know what the 1. most "kedro-friendly" way 2. easiest way would be, since the method in the issue I linked would require us to perform changes in two separate catalogs if anything changes, but the custom runner requires more development, and additional testing

@antony.milne is this correct, I'm pretty sure DryRunner executes, just doesn't save

I think a combination of this DataCatalog method and after_catalog_created hook might be what we're looking for

I might be missing something but the implementation in that example and its description both point to it only listing the nodes without actually running them

No, it doesn't execute. From the docstring itself. > """``DryRunner`` is an ``AbstractRunner`` implementation. It can be used to list which > nodes would be run without actually executing anything. > """

ah my mistake

Hello, I've run into a wierd issue with partitioned datasets on aws s3, for some reson kedro adds a none existing file to the load dict (with key ''), this causes a load error. A simple key check solves it but I rather solve it properly. Any ideas on what might be causing this? {'': <bound method AbstractVersionedDataSet.load of >, '20220609112232.csv': <bound method AbstractVersionedDataSet.load of >}

Hey does anyone know how to add credentials for pandas big query? https://github.com/kedro-org/kedro/blob/develop/kedro/extras/datasets/pandas/gbq_dataset.py I am trying to add the service account file to credentials.yml (not having much luck)

It should have the same API as pandas, could you connect to the data with just

pd.read_gbq

?

Does the doc helps? https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.GBQTableDataSet.html https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_gbq.html

question: catalog.add() seems not change local config.file Description: 1. jupyter notebook import the existing catalog.yaml to load all data (works) 2. use catalog.add and save the a new dataframe (works, data in catalog.list() and DF can be found on disk.) 3. check the local config.yaml file, cannot find the item