want to route data folder filesin django(want to read data folder files of kedro in django rest api)

sure

So you can: - Talk to an API via

APIDataSet

which is a thin wrapper of the

requests

library - Read filepaths in most of the other datasets - Read SQL via Pandas or Spark But there isn't any formal integration with Django out of the box

ok....is there any example code available ?

I would encourage you to run through the Spaecflights tutorial, if this is your first time using Kedro https://kedro.readthedocs.io/en/stable/tutorial/spaceflights_tutorial.html

Hi everyone, I have a pipeline (Current_Kedro_Pipeline.png) that is pulling from an SQL table and then filtering by a start and end datetime. However I don't really want to pull the entire table just in order to immediately filter it. So I'm wondering: - What is the best way to create an SQL dataset that accepts inputs which you can then use to query? - I see that there is a "load_args" option to pass in data to a dataset but it doesn't seem as though this can be used as options for the query string itself (please do correct me if I'm wrong about this)? - Therefore I'm wondering if the best alternative is to created a "CustomDataset" that utilises raw SQLAlchemy? - Additionally, I don't know how feasible this is but one of the things I really like about Kedro is the traceability of the datasets. However, If I solve the aforementioned problem there will then be no association between "Patient Session Datetime Filtered" and the "Patient Sessions" dataset. I'm wondering if it would be possible to create manual links between datasets to show if one is derived from another, just to keep that traceability? I mocked up an example using a dotted line (Kedro_Pipeline_Dependant_Database_Example.png) to try and illustrate what I mean. Any help is much appreciated! Thank you, Lawrence

So the way to do this is to pass dummy data between the nodes, they don't need to have catalog entries but will drive the execution order too

Thank you, could you possibly elaborate a bit more? Or point me to an example that does this? I don't really know how passing dummy data between nodes is achieved.

Hi, I had to do something like that in the past. To reuse the SQLDataSet of Kedro, I split the query in two steps. In the first, I was using a TextDataset with parameters that were replaced in one node (for example with a .format). In the second, the node had the filled SQL query and the connection string as inputs, was instantiating a SQLDataSet and returning the result of the load. This node is in fact generic. I liked this implementation because it was very flexible and allowed me to put all my SQL queries in a dedicated folder and to use a linter for the queries.

@PetitLepton @LawrenceS this is very much possible from a custom implementation point of view. The philosophical reasons why we don't make this native functionality are as follows: https://github.com/kedro-org/kedro/issues/904#issuecomment-925969655

Perhaps as more dataframe like abstractions for SQL engines i.e. Snowpark we will be able to approach things that way

Hiya- using kedro to try to visualize data pipelines w/ dask and ran into a small hiccup- setting up dataframes and running them locally is great, but when I want to use something like coiled, i'm a little confused as to how to let kedro know I'd like to run the coiled init function before a pipeline starts running. Could I get some pointers?

You could adapt how our Spark set up works https://github.com/kedro-org/kedro-starters/blob/main/pyspark/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/context.py In fact @deepyaman expanded on this in these docs recently https://kedro.readthedocs.io/en/latest/deployment/dask.html

is context.py the only entry point needed?

e.g. if I hardcode credentials there, will I have something that will run?

I think so - but I don't endorse hardcoding creds!

I know, I just need something running asap

it doesn't look like it's running the context.py file

reload_kedro is giving me

TypeError: __init__() missing 2 required positional arguments: 'config_loader' and 'hook_manager'

is this some newbie mistake I'm making?

oh alright then

I just removed the kwargs from the constructor and it worked

it works!

tysm

Amazing!

Hello all! When attempting to run

kedro viz

I get the attached error. The kerdro versions I'm running are below and I did

pip install kedro[all]

however, still experience the error. kedro==0.18.2 kedro-telemetry==0.2.1 kedro-viz==4.7.1

Disregard, it was a typo needed to capitalize the "S" in DataSet😅

You got it right!

is there a way to make a variable declared in a ProjectContext accessible during runtime? I noticed that in the documentation all of the parameters provided (https://kedro.readthedocs.io/en/latest/kedro.framework.context.KedroContext.html#kedro.framework.context.KedroContext) are read only, but I need to pass a reference to my session object to code in nodes in my pipeline...