Hello I am trying to figure out two

Thanks for the tip. Just need the params for this use case actually. Basically, there’s a node that will split the data based on some in-scope brands defined in params. Hence, the node output would ideally be dynamically defined based on those in-scope brands.

Could not pass that as a dataset?

I think that might be a better idea. Something like an hdf file. Thanks.

Hey, I'm getting a lot of DeprecationWarnings from one of my imported libraries. How to suppress them? import warnings doesn't work when running with kedro

What version of kedro? For most versions you simply need to go to logging.yml where weve exposed logging.dictConfig as yaml

0.17.7, will look inside logging.yml, thanks

hi everyone,i have to implement kedro with django.i have to route kedro data folder data in django and use that for web application ,i m not getting anything anywhere.pls help me with code,video link or any document link or with anything.

What are you struggling to do?

can you also please not post on every channel

/ remove your other posts

want to route data folder filesin django(want to read data folder files of kedro in django rest api)

sure

So you can: - Talk to an API via

APIDataSet

which is a thin wrapper of the

requests

library - Read filepaths in most of the other datasets - Read SQL via Pandas or Spark But there isn't any formal integration with Django out of the box

ok....is there any example code available ?

I would encourage you to run through the Spaecflights tutorial, if this is your first time using Kedro https://kedro.readthedocs.io/en/stable/tutorial/spaceflights_tutorial.html

Hi everyone, I have a pipeline (Current_Kedro_Pipeline.png) that is pulling from an SQL table and then filtering by a start and end datetime. However I don't really want to pull the entire table just in order to immediately filter it. So I'm wondering: - What is the best way to create an SQL dataset that accepts inputs which you can then use to query? - I see that there is a "load_args" option to pass in data to a dataset but it doesn't seem as though this can be used as options for the query string itself (please do correct me if I'm wrong about this)? - Therefore I'm wondering if the best alternative is to created a "CustomDataset" that utilises raw SQLAlchemy? - Additionally, I don't know how feasible this is but one of the things I really like about Kedro is the traceability of the datasets. However, If I solve the aforementioned problem there will then be no association between "Patient Session Datetime Filtered" and the "Patient Sessions" dataset. I'm wondering if it would be possible to create manual links between datasets to show if one is derived from another, just to keep that traceability? I mocked up an example using a dotted line (Kedro_Pipeline_Dependant_Database_Example.png) to try and illustrate what I mean. Any help is much appreciated! Thank you, Lawrence

So the way to do this is to pass dummy data between the nodes, they don't need to have catalog entries but will drive the execution order too

Thank you, could you possibly elaborate a bit more? Or point me to an example that does this? I don't really know how passing dummy data between nodes is achieved.

Hi, I had to do something like that in the past. To reuse the SQLDataSet of Kedro, I split the query in two steps. In the first, I was using a TextDataset with parameters that were replaced in one node (for example with a .format). In the second, the node had the filled SQL query and the connection string as inputs, was instantiating a SQLDataSet and returning the result of the load. This node is in fact generic. I liked this implementation because it was very flexible and allowed me to put all my SQL queries in a dedicated folder and to use a linter for the queries.

@PetitLepton @LawrenceS this is very much possible from a custom implementation point of view. The philosophical reasons why we don't make this native functionality are as follows: https://github.com/kedro-org/kedro/issues/904#issuecomment-925969655

Perhaps as more dataframe like abstractions for SQL engines i.e. Snowpark we will be able to approach things that way

Hiya- using kedro to try to visualize data pipelines w/ dask and ran into a small hiccup- setting up dataframes and running them locally is great, but when I want to use something like coiled, i'm a little confused as to how to let kedro know I'd like to run the coiled init function before a pipeline starts running. Could I get some pointers?

You could adapt how our Spark set up works https://github.com/kedro-org/kedro-starters/blob/main/pyspark/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/context.py In fact @deepyaman expanded on this in these docs recently https://kedro.readthedocs.io/en/latest/deployment/dask.html

is context.py the only entry point needed?

e.g. if I hardcode credentials there, will I have something that will run?

I think so - but I don't endorse hardcoding creds!

I know, I just need something running asap

it doesn't look like it's running the context.py file

reload_kedro is giving me

TypeError: __init__() missing 2 required positional arguments: 'config_loader' and 'hook_manager'