This was my hack to expose the globals to the jinja2:``` class TemplatedConfigLoaderGlobals(TemplatedConfigLoader): def _load_config_file(self, config_file: Path) -> Dict[str, Any]: """Load an individual config file using

anyconfig

as a backend. Args: config_file: Path to a config file to process. Returns: Parsed configuration. """ # for performance reasons import anyconfig # pylint: disable=import-outside-toplevel if "globals" in str(config_file): globals = {} else: globals_pattern = "*globals.yml" globals = self.get(globals_pattern) if globals_pattern else {} return { k: v for k, v in anyconfig.load( config_file, ac_template=True, ac_context=globals ).items() if not k.startswith("_") } ```

@User very nice is it working?

The globals are not exposed to configfiles that defines globals as that gave an infinite recursion which is why I added

if "globals" in str(config_file):

This is however not folowing the globals_pattern so it is a bit of a hack, but it works for my project at least

Really nice work - I'm making a note on how we support this natively

If I have my way we'd use Jsonnet instead of Jinja https://jsonnet.org/

Something about using Jinja with a whitespaced language like YAML gives me the ick

Hi everyone. I am new at this community. So that I don't know this feature. Is Kedro support Apache Hive?

Here are the out-of-the-box datasets: https://kedro.readthedocs.io/en/latest/kedro.extras.datasets.html Including: kedro.extras.datasets.spark.SparkHiveDataSet

Thanks @User - yes there is Hive support via the

spark.SparkHiveDataSet

Thanks @User and @User 🙂 Nice to meet you

Hello team!! Is there any way we could integrate it with DBT framework?

High I’ve been putting some thought into it - today it’s simply a matter of declaring Kedro datasets for the dbt models you want to read from. A question for you - how would you want it to work?

Is there a way to prioritise modular pipelines ? fetching data then preprocessing data is the right order, but the project try to convert the model before the training pipeline. Maybe it's linked to the fact training pipeline and converting pipeline have

None

as outputs ?

So you can either make them explicit dependencies (easiest) or get fancy with you CLI commands

kedro run --pipeline preprocessing && kedro run --model_train

will run sequentially, or orchestrate these explicitly in a higher level tool like Airflow/Prefect etc

explicit dependencies

are done with kedro directly in

pipeline_registry

?

Because I read this

The order in which you add the pipelines together is not significant and data_science_pipeline + data_processing_pipeline will result in the same pipeline, since Kedro automatically detects the correct execution order for all the nodes in the resulting pipeline.

in the spaceflights tutorial, but I don't know how it's done actually 🙂

Ah what that means is that Kedro works out the execution order via the node inputs and outputs.

By explicit dependencies I mean you make sure you use a dataset outputted from pipeline 1 in pipeline 2

thats the way Kedro knows how to make it happen in the right order

it's called topological sorting

Ok but in my case the 1st pipeline has no output because pytorch lightning handles alone the model checkpoint save, and the 2nd pipeline has also no output because onnx handles the model conversion alone. They both use a variable written in

parameters

, Maybe they could output something useful.

I mean if I add a parameter as output that is required (something like

training_done=True

) in the second pipeline, that could work ?

It absolutely would, it doesn't technically have to be a meaningful connection!

it worked fine

Thanks for help !

Hi, We are currently using Kedro to build our pipelines to build our ML models. The kedro documentation explains how to deploy a Kedro pipeline on AWS Step Functions where every Kedro node is an AWS Lambda function or running the entire pipeline in Sagemaker. But our team wanted to deploy the pipeline to AWS and have some nodes run as a lambda ,some nodes(like the node that trains the model) as a sagemaker train job, some nodes that take a long time to run in a ECS/Sagemaker process job. Our team wrote a plugin which manages to do this. The developer just adds a tag to a node(like "lambda"/"ECS"/"Sagemaker_train"). After that if the dev runs a command thats added by the plugin we parse the pipeline and based on the tags we use CDK to deploy the pipeline. The plugin has the CDK code needed to then deploy the pipeline on AWS Step functions Before the team puts more effort into optimizing/adding more features to the plugin i wanted to check if a solution already exists to do this. And if our approach to deploy the Kedro pipeline to AWS is correct.

To my knowledge this doesn't exist - but I would say this is a really neat idea. If you wanted to brainstorm some ideas on the implementation I'm sure some of the maintainers could help think through it

@User the other thought is running a set of nodes via

kedro run --pipeline={name}

Custom Node Execution Target