Thanks @avan-sh , is this by design? then can I export the catalog in jupyternotebook to a local datacatalogYAML file? I dont think copy the path and dataset name and paste is a good practice or the only approach.

Are you trying to create catalog interactively? It is quite difficult to infer the yaml file base on the python object, there are many possible options

yeah. that's why I am asking. I am a little surprised we can not update catalog yaml interactively. can you elaborate a little about other options?

write catalog info to files

Hello everyone, has anyone a good resource on how to use great expectations with kedro?

Using Great Expectations

Is there a discussion already open for how heavy-handed kedro is on it's requirements? I have some issues on occasion where things have moved since the last kedro release, but kedro does not allow new enough versions. currently my build-reqs is broken because of this issue on pip-tools. https://github.com/jazzband/pip-tools/issues/1617 It's fixed after 6.5.1, but kedro does not want me to use it (I am able to, but get pip warnings all over). Will raise an issue if there isn't already one somewhere.

kedro requirements

How to get a list of pipelines (in a CLI utility)? In the past (<1.18) I did something like this:

metadata = bootstrap_project(project_dir)

with KedroSession.create(metadata.package_name) as session:
    context = session.load_context()
    pipeline_names = sorted(context.pipelines.keys())

Now, however, context.pipelines does not exist anymore. What is the proper way to do this? TIA

That should be

from kedro.framework.project import pipelines

The reason this is now possible as in 0.18 pipelines are basically python packages and don't need any magic to be registered

@datajoely @antony.milne Antony's solution works for me, since I really needed the names under which the pipelines were registered, not the package. Later on, I do a session.run(pipeline_name=pipeline_name)

hi everyone, i have a question about the project structure. if i want to separate the core logic and the kedro pipelines into 2 modules, is there any configuration that i need to modify correspondingly? file structure (omitting all other files): - my_project - src - core - kedro

You don’t need change any configuration for this, your pipelines should run fine irrespective of where you define them as long as they’re imported & registered under register_pipelines

got it, thanks. i was just wondering if there will be issues with packaging/Kedro's load_context..

It should be fine as long as you’re not moving settings & pipeline_registry files.

So

core

is some Python module and

kedro

is where the kedro project is? This is fine as

core

is external to kedro as any

site_pacakge

you have.

kedro

is really where the kedro project is so you just doing

kedro xxx

from there.

> So core is some Python module and kedro is where the kedro project is? yes, you are correct. i tried it out anyway and i found that i need to modify the

pyproject.toml

as follows:

[tool.kedro]
package_name = "kedro"
project_name = "my_project"
project_version = "0.17.5"

Let's say I just cloned my kedro project repo to another machine, and its datasets are versioned and configured to use S3 for storage. If I try to run a pipeline that depends on those datasets I get the infamous

kedro.io.core.VersionNotFoundError

. Bucket has versions all the way up to

2022-06-07T22.04.39.460Z/

and the error says

2022-06-23T16.20.52.945Z

. Is this the intended behavior? Thanks

How does the file look like before your changes?

What is the command that you run? Did you specify any version? If not it should just grab whatever latest version you have in the S3 store

I'm trying to upgrade from v0.17.4 to v0.18.1, and when I try to do kedro run, it's still trying to look for kedro.versioning

What is the error exactly? If it's something about

journals

I suspect it's because it's still mentioned in your logging.yml file. Easiest way to fix that is just to copy and paste this into your logging.yml file: https://github.com/kedro-org/kedro/blob/0.18.1/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml

Yes, it was indeed about some journal logger. Will try that. Thanks!

Hello. Where would I go looking to find where the params file is being read from disk? I'm having an issue with non-ASCII characters on Windows. I am saving my params.yml as UTF-8 encoding but python, taking locale.getpreferredencoding(), is attempting to read it using CP1252 which generates gibberish for characters like ñ, é etc. If I wanted to change the config of a YAMLDataSet I would change it like here: https://github.com/kedro-org/kedro/issues/772#issuecomment-847650332 but I don't know if there is a similar config for the parameters.yml file.

params file encoding

Sorry for the late response. I just tried

kedro run

, no version specified.

Hello, running a Kedro pipeline results in my job getting killed. It looks it is running out of memory as it is trying to save the result of a node to a ParquetDataSet. My inputs are about 2G in size. Any solutions or suggestions?

So it would be great to get a stack trace to understand what's going on. Could you try wrapping your

ParquetDataSet

in a

PartitionedDataSet

would allow you to write smaller chunks and won't fail