Kedro is an open-sourced Python framework for creating maintainable and modular data science code.

Kedro

i was waiting for connection to work before working on pipelines haha

so you can do that if you haven't registered the pipeline in `pipeline_registry.py` I think

but if it's already registered it needs to resolve

if this is your first time - it may be worth running through the spaceflights tutorial
https://kedro.readthedocs.io/en/stable/03_tutorial/01_spaceflights_tutorial.html

or potentially follow along this training session I did a couple of weeks ago 
https://www.youtube.com/watch?v=NU7LmDZGb6E&t=2191s

and tried the same thing in another repo with that

if you're able to share the code I can get it working?

let me play around with some of the tools you've given

but I can do it some point in the next 24 hours

and see if i cant resolve, i'll look into the no nodes pipeline bit and see if it changes it

and i'll cross my T's and dot my I's on the spaceflight as well 🙂

okay in `pipeline_registry.py` you can just remove the keys from the dictionary

I'm going to create a thread so capture this topic though

so i rebuilt the spaceflgiht tutorial again, and the excel files passed fine

```Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.32.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.32.0
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
shuttles = catalog.load("shuttles")
Traceback (most recent call last):
  File "C:\Users\alexr\kedro-repos\masterenv\master-kedro\lib\site-packages\IPython\core\interactiveshell.py", line 3524, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-7b32a5522d82>", line 1, in <cell line: 1>
    shuttles = catalog.load("shuttles")
NameError: name 'catalog' is not defined```

i comment out the sql from catalog and credentials, and get the same name error

If you try and kedro run what error do you get?

```In [3]: kedro run
  File "<ipython-input-3-283f190c240f>", line 1
    kedro run
          ^
SyntaxError: invalid syntax


In [4]: 
```

did a reload and back to no pipelines, with ipython not loading catalog or context whiel our SQL is in there

Sorry you need to do that outside of ipython

sorry it is installing reqs, it'll be a moment

the behavior seems to be, if sql in yml, then "no nodes"

if sql not in yml, and reload, then it loads context/catalog etc

Pyodbc needs a bunch of things available in the environment and we just need to make sure it's not that

ive watched so much waylon walker and dataengineerone

going to try to document the steps as i remember them

But I'm now going to make a note to look into the no nodes issue

i think you guys do a great job, what this framework unlocks is so tantalizing

my company is moving to databricks in the next month or two and i loved the idea of this integrated with DBs

There was a slight change to the SQL datasets in 0.17.7 where we only make a single connection to database whereas we used to create a new one for each operation

So maybe we are swallowing the requirements error

Good luck! Shout if you have any questions

and if i figure it out i'll report back 🙂

plugging right along 🙂 is there a preferred method for saving 3d numpy arrays to be called later in the pipeline?

`PickleDataSet` is probably the best shout - potentially with the `joblib` backend

also note you don't have to save if you're using things later in the pipeline as it will automatically be handled as a `MemoryDataSet`