Hello all. Newbie to kedro here. I'm currently testing out using kedro for my organisation which uses Qubole for most of our ml work. In the Qubole environment we have access to a plain jupyter notebook where we are able to read files from s3 and run our models using the clusters available. I've attempted to load my kedro project as a folder in s3 and run the project from my notebook but cant seem to get the code to work. I believe it has to do with referencing the path in which the kedro project is location in s3. Attached is the code used to run in jupyter. Do you have any recommendations on how i can get this working ? Here's the error im getting * Could not find the project configuration file 'pyproject.toml' in //s3:/*

Hi Aanan, I'm afraid I'm not familiar with Qubole but I'm wondering if you could use your kedro project as a package. I.e. package up your project as a python package, pip install it and run it with

my_package

or

python -m my_package

. And you were right, the error message suggests it's not finding the file that distinguishes a regular folder from a kedro project, namely

pyproject.toml

.

Another thing to try would be to see if you can use line magics to sneakily run the kedro project that way. e..g using the system line magic

!

which runs a command (so you do

!kedro run

which works in my test project). Or if that fails maybe you could use

%run

and point to the kedro project

__main__.py

file. https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-run