@User did you get it to work?

Greetings ! I'm facing a very similar issue to this one that happened in an older version https://github.com/quantumblacklabs/kedro/issues/291 Basically I can't save my dataset due an encoding error

kedro.io.core.DataSetError: Failed while saving data to data set CSVDataSet(filepath=C:/Users.........csv, load_args={'encoding': utf-8}, protocol=file, save_args={'index': False}). 'charmap' codec can't encode character '\x91' in position 9: character maps to

I'm on kedro 0.17.4 python 3.7.9 windows 10 pro

@User , I was never able to figure it out. @User, I think you might be miss-understanding what I was asking. The video above and documentation seem to only reference creating a single custom environment. I am talking about inheritance from the various environments. For example, right now, if I run my dev environment, I also inherent from base if dev does not include the save parameter/catalog. I am wanting to take it a step further, where there are three levels of inheritance. Specifically such that temp inherits from dev which inherits from base (base -> dev -> temp).

Hi @User! I have defined similar conf inheritance patterns in the past by modifying the

register_config_loader

hook under

src/<package_name>/hooks.py

. __**Example**__

from pathlib import Path
class ProjectHooks:
    """Project hooks."""

    @hook_impl
    def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
        # Force local as ultimate overriding params, regardless of env chosen.
        conf_paths.append((Path(CONF_ROOT) / "local"))
        return TemplatedConfigLoader(conf_paths, globals_pattern="*globals.yml")

In this case, the inheritance will be

base

->

<custom_env>

->

local

. You can append other envs to

conf_paths

to customize this behavior.

@User, I just found your project find-kedro. Cool stuff. Quick question: why do you utilize the full path for the pipeline name (for example src.default_kedro_159.pipelines.datas_science.pipeline) rather than just grabbing the directory name (in this case data_science), such that when running the project you just need "Kedro run --pipeline=data_science"?

@User , It was mostly so that it would be most broadly useful. I cannot guarantee how others structure your project. The roots of find-kedro comes from my experience on

0.14.x

project that had a very deeply nested set of pipelines. You can easily change up the keys in your register pipelines module. You can make up rules that make most sense for your team. The core of what

find-kedro

does to create pipelines based on name (quite similar to how pytests picks up tests) will still be there, you will just make a slight tweak to make it cleaner for your project.

@User Please let me know if you decide to use it and have any thoughts on it, I am not sure if anyone else is using it, but its the first thing I put on most of my projects.

@User, I am planning on using it on a new project, then editing the output such that the key just requires the pipeline directory name aka "data_science". We did something similar to your solution on another project, but your's is one line of code to implement rather than the 10 we needed before.

+ ours was only for pipelines, not nodes and everything else.

My pattern is to just put everything in a list of nodes, we really struggled onboarding early on. The docs were more sparse, and we had a very deeply nested setup. I spent way too many hours helping wtih packaging details just to get nodes to show in pipelines.

Hi all :). I'm currently testing out Kedro, and have made a small example with a pipeline training a deep learning model on MNIST. The pipeline is composed of three smaller pipelines, with the first of these consisting of a seed_everything() node, which seeds all random generators. However, when I run the entire pipeline, it does not run the three smaller pipelines sequentially, even though I'm using a SequentialRunner. It seems like the order is based on the data dependencies, rather than the order defined in pipeline_registry(). Is there a way to ensure that a sub-pipeline finishes before starting the next when defining a pipeline?

Execution order

Do people usually create/edit data catalog YAML file using python in their pipelines? I have a pipeline with known number of output datasets with consistent meaning and naming (

file_A

,

file_B

,

file_C

). I want the folder that this pipeline runs in have its own dynamically generated data catalog so other people can go in and inspect the results from the pipeline easily by just taking the example from https://kedro.readthedocs.io/en/latest/05_data/01_data_catalog.html#configuring-a-data-catalog , is it possible to do this:

python
io = DataCatalog(
    {
        "bikes": CSVDataSet(filepath="../data/01_raw/bikes.csv"),
        "cars": CSVDataSet(filepath="../data/01_raw/cars.csv", load_args=dict(sep=",")),
        "cars_table": SQLTableDataSet(
            table_name="cars", credentials=dict(con="sqlite:///kedro.db")
        ),
        "scooters_query": SQLQueryDataSet(
            sql="select * from cars where gear=4",
            credentials=dict(con="sqlite:///kedro.db"),
        ),
        "ranked": ParquetDataSet(filepath="ranked.parquet"),
    }
)

and then do

io.to_config()

? we have

io.from_config()

but not

io.to_config()

to generate YAML file from the data catalog object

@User , what do you mean by dynamic catalog? You have a known set of outputs, why does it need to be dynamic? I have definitely done some hacky python scripts to generate yaml.

I have had a few that literally just print the config, and I run

python gen_catalog.py > catalog.yml

, End of the day yaml is still all that goes into the project and it's just a quick shortcut for me to generate a bunch of entries quick.

sorry for the confusion, I meant catalog that is generated using python code

@User do you have script that can be used to turn a

DataCatalog

like the example above into yaml? Would greatly appreciate it!!!

ah, I see. I do not have one no.

It defeinitely seems possible

Also see [here](https://discord.com/channels/778216384475693066/877970253450199160/877977652198264962). "Dynamic data catalog" could mean using python to export a

DataCatalog

object yaml, and parameterize the data catalog generating script to generate different number of datasets (potentially with no intersection between the two sets) in different environments

Yep! I just thought that someone must have done it. Now I'm confident that everyone is mainly just manually editing the yaml files when they want to add datasets (or maybe use

kedro create_catalog

as a starting point then manually edit) 🙂

Actually I have never seen anyone make a catalog like you pulled from the docs using the python api (well except Lim). Personally I like it, but it just doesn't seem well supported. I raised an issue a while back that the datasets should document the yaml api rather than python api. I think it had good response, its just not a priority.

I'm only using kedro catalog to combine with Prefect and not using Kedro pipelines, that's why I'd need this 🙂 . However, I still think things like this should be more supported for when people start with the

mini-kedro

minimal starter and have not needed pipelines yet

@User I just whipped up the miinimal code snippet to go from the Dataset Class object to yaml string representation. Will need to modify for custom Datasets, but it's a start! 🙂

python
import inspect

from kedro.extras.datasets.pandas import CSVDataSet

full_class_name = inspect.getclasstree([CSVDataSet])[-1][0][0] #kedro.extras.datasets.pandas.csv_dataset.CSVDataSet

full_class_name_str = str(full_class_name) #<class 'kedro.extras.datasets.pandas.csv_dataset.CSVDataSet'>

yaml_class_str = full_class_name_str.partition("kedro.extras.datasets.")[-1].strip("'>") 

print(yaml_class_str) #pandas.csv_dataset.CSVDataSet

In a similar line to the dynamic catalog, has anyone done dynamically generated nodes inside of a pipeline based on params in the catalog? I am utilizing a for loop based on a list in the yaml to generate nodes. However, to do this I created a basic function which acquired the params from the context. However, when pip-installing a project and running a pipeline from the pip-install, this method seems to fail due to the

__main__.py

file attempting to configure the pipelines prior to the session setting the Context with

configure_project(Path(__file__).parent.name)

. Because the data files are not necessarily saved within cwd it is causing the pipeline registration to fail. I tried to create a ConfigLoader with the correct location if a session wasn't present, but this just seems to make the entire pipeline hang. Any idea how to get around

configure_project

?

So I'm afraid

configure_project

can't/shouldn't be bypassed as that's where settings and pipelines are (lazily) configured, in order to a) be able to import them anywhere in a project, and b) use them in the framework code. If you really really need the parameters, I suggest recreating the configloader logic of fetching the parameters in a helper function that you can call in the node. But generally dynamically generated pipelines are to be avoided if you can, I'm curious what your use case is maybe there's an alternative?

Dynamic Pipelines.

Hi, I am very new to this. I might be missing something but it seems we can only input as raw data file by file. In the catalog, each entry seems to be only one file. However, my raw data is an entire directory from which I need to load data individually. I was wondering if there was a way to pass into a pipeline a directory as inputs instead of specific catalog entry that are each related to a file? Can we have in the data catalog a directory instead of a file? Sorry if this seems completely obvious...