Thanks @datajoely ! I think that lists all the nodes without actually running them, but maybe we could adapt it to replace certain tagged nodes' input/output. Though I don't know what the 1. most "kedro-friendly" way 2. easiest way would be, since the method in the issue I linked would require us to perform changes in two separate catalogs if anything changes, but the custom runner requires more development, and additional testing

@antony.milne is this correct, I'm pretty sure DryRunner executes, just doesn't save

I think a combination of this DataCatalog method and after_catalog_created hook might be what we're looking for

I might be missing something but the implementation in that example and its description both point to it only listing the nodes without actually running them

No, it doesn't execute. From the docstring itself. > """``DryRunner`` is an ``AbstractRunner`` implementation. It can be used to list which > nodes would be run without actually executing anything. > """

ah my mistake

Hello, I've run into a wierd issue with partitioned datasets on aws s3, for some reson kedro adds a none existing file to the load dict (with key ''), this causes a load error. A simple key check solves it but I rather solve it properly. Any ideas on what might be causing this? {'': <bound method AbstractVersionedDataSet.load of >, '20220609112232.csv': <bound method AbstractVersionedDataSet.load of >}

Hey does anyone know how to add credentials for pandas big query? https://github.com/kedro-org/kedro/blob/develop/kedro/extras/datasets/pandas/gbq_dataset.py I am trying to add the service account file to credentials.yml (not having much luck)

It should have the same API as pandas, could you connect to the data with just

pd.read_gbq

?

Does the doc helps? https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.GBQTableDataSet.html https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_gbq.html

question: catalog.add() seems not change local config.file Description: 1. jupyter notebook import the existing catalog.yaml to load all data (works) 2. use catalog.add and save the a new dataframe (works, data in catalog.list() and DF can be found on disk.) 3. check the local config.yaml file, cannot find the item

That is definitely the expected outcome. Catalog object available from jupyter notebook is only for development and any additions from notebook are not reflected automatically in local file. You'll have to manually add the config to catalog.yaml or parameters.yaml file.

Thanks @avan-sh , is this by design? then can I export the catalog in jupyternotebook to a local datacatalogYAML file? I dont think copy the path and dataset name and paste is a good practice or the only approach.

Are you trying to create catalog interactively? It is quite difficult to infer the yaml file base on the python object, there are many possible options

yeah. that's why I am asking. I am a little surprised we can not update catalog yaml interactively. can you elaborate a little about other options?

write catalog info to files

Hello everyone, has anyone a good resource on how to use great expectations with kedro?

Using Great Expectations

Is there a discussion already open for how heavy-handed kedro is on it's requirements? I have some issues on occasion where things have moved since the last kedro release, but kedro does not allow new enough versions. currently my build-reqs is broken because of this issue on pip-tools. https://github.com/jazzband/pip-tools/issues/1617 It's fixed after 6.5.1, but kedro does not want me to use it (I am able to, but get pip warnings all over). Will raise an issue if there isn't already one somewhere.

kedro requirements

How to get a list of pipelines (in a CLI utility)? In the past (<1.18) I did something like this:

metadata = bootstrap_project(project_dir)

with KedroSession.create(metadata.package_name) as session:
    context = session.load_context()
    pipeline_names = sorted(context.pipelines.keys())

Now, however, context.pipelines does not exist anymore. What is the proper way to do this? TIA

That should be

from kedro.framework.project import pipelines

The reason this is now possible as in 0.18 pipelines are basically python packages and don't need any magic to be registered

@datajoely @antony.milne Antony's solution works for me, since I really needed the names under which the pipelines were registered, not the package. Later on, I do a session.run(pipeline_name=pipeline_name)

hi everyone, i have a question about the project structure. if i want to separate the core logic and the kedro pipelines into 2 modules, is there any configuration that i need to modify correspondingly? file structure (omitting all other files): - my_project - src - core - kedro

You don’t need change any configuration for this, your pipelines should run fine irrespective of where you define them as long as they’re imported & registered under register_pipelines

got it, thanks. i was just wondering if there will be issues with packaging/Kedro's load_context..

It should be fine as long as you’re not moving settings & pipeline_registry files.

So

core

is some Python module and

kedro

is where the kedro project is? This is fine as

core

is external to kedro as any

site_pacakge

you have.

kedro

is really where the kedro project is so you just doing

kedro xxx

from there.

> So core is some Python module and kedro is where the kedro project is? yes, you are correct. i tried it out anyway and i found that i need to modify the

pyproject.toml

as follows:

[tool.kedro]
package_name = "kedro"
project_name = "my_project"
project_version = "0.17.5"