I have now tried using astronomer to get kedro project into airflow following this https://kedro.readthedocs.io/en/stable/deployment/airflow_astronomer.html#deployment-process

this is how it looks now in airflow, kedro module not found

This means you haven't installed requirements on the server

Well!! i have airflow, kedro and astro installed locally and im doing all of this locally for testing

so which server would u mean

Or at least the virtual environment that airflow is looking for

and the conf/ is at

/Users/sl/airflow/

where the dags folder in airflow.cfg is defined

dags_folder = /Users/sl/airflow/dags

and i still have the issue

raise ValueError(ValueError: Given configuration path either does not exist or is not a valid directory: /conf/base

. i have read the docs and tried several thing but couldn't succeed. Would be glad if i can get some help here

Hi @datajoely, how are you? Just wanted to ask if you have any resource/references which can help me in leveraging kedro functionalities inside Databricks multi node cluster.

So we're working on this as we speak. For now the guide here: https://kedro.readthedocs.io/en/stable/deployment/databricks.html

and some of the mentions in the release notes: https://github.com/kedro-org/kedro/blob/main/RELEASE.md

are the best bet, but if you have anything specific we can assist

For now I'm just doing my research. Nothing specific at the moment. Will let you know. Thanks a ton 😁

No worries! Expect it to get better in the next release too

Sure thing.

@datajoely should we wait to hear if any debugging is going on at ur end to suggest any fix? or is the issue still sound to be specific for us?

This isn't an issue - configuration isn't packaged by design you need to copy it over yourself

We follow principles of the 12 factor app which mandates this

i tried to describe it in git, where we copied the conf/ folder and files ...yet it persists...wonder what would be right path for them to be copied?

Hi @Deep , been working on databricks setup. Happy to help if you have any issues

Hi @avan-sh thats very kind of you buddy. Sending you a fr

Did anyone here setup airflow instance using kedro-package pipeline? unable to configure, spent good amount of time following the docs, would be great if anyone walked through the same path can assist. The issue is summarised here https://github.com/kedro-org/kedro-plugins/issues/46

Kedro-airflow issue cannot find configu...

Kedro template configuration does not load globals.yml configuration into catalog.yml https://stackoverflow.com/questions/73081209/kedro-template-configuration-does-not-load-globals-yml-configuration-into-catalo

Hi there. I created a custom dataset to manage some spark delta tables. One thing I need in order to save the data is to receive the replace where query. Is there any way to send this info plus the data from the previous node execution or recover this information inside the custom dataset? The reason why I can’t use the parameters file in the usual way is because I need values defined in execution time

Kedro template configuration does not load/parse variables https://stackoverflow.com/questions/73105524/kedro-template-configuration-does-not-load-parse-variables

Hi @avan-sh, been trying to reach out to you. Please dm me as I have some queries. Take your time, no hurry.

Hi all, I am using Kedro v0.18.2 and I have a question. When I want to config the catalog I got an error (Error.png). The error is in the globals.yml (globals yml.png) where I specified the dataset that is initialised in my folder 'base' (__init__ from base). When the file, where I config the catalog, is a folder level above 'base' it works, but when the file (nodes.py) is on a lower level (Folder structure) it doesn't work. I tried to specify the whole path in the globals.yml, but that doesn't work. I hope you can help me out!