kedrosequential_runner.py at main · ked...

Custom DataSet for larger than memory data -- dask, SQL, other?

How to run a kedro pipeline interactively like a fuction https://stackoverflow.com/questions/73936203/how-to-run-a-kedro-pipeline-interactively-like-a-fuction

I have achieved this using sklearn - I just pass the object name as a string in the parameters.yml and then I parse it with getattr() to get the specific object

I just have to import the module first

Lets say for simplicity, I have a pipeline with a single node, which inputs dataset A and output dataset B. How to handle a situation where I want to run this pipeline each day (with airflow for example) and I want dataset B of the previous day as input (as dataset A) in the new day? In the ideal scenario I'd like to have a single versioned dataset, where I overwrite the input dataset with a new version. However, kedro doesnt allow the same dataset as input and output in a single node.

This is the default of kedro version dataset, If you don't specified the version it will fetch the latest. Overwriting inputs is not a good idea because you risk getting a corrupted file, the whole point of orchestrator is handling the dependencies and retry logic, and you can't have that if you overwrite the data.

Makes sense and I agree, however how do you setup the data catalog if the input file should be the same as the output file, since it is dependent on the output of the same node (but executed a day before) and the data catalog is time-independent.

You can have two catalog entries pointing to the same location. I strongly advise not to do this as there are likely better solution.

That is indeed what I'm currently doing, however indeed doesnt seem the correct solution.

sounds like what you're doing is taking yesterday's dataset, adding new data, and saving it again You are mutating your raw data which is not a good idea, you could corrupt it if your code crashes. If the overhead is not big, I would save each day's data into a separate timestamped file, then use kedro with an aggregation node that merges all the files into one dataset (by using a PartitionedDataset for example). The timestamped file doesn't have to be made with kedro and you can consider it as immutable and back it up with ex. S3 versioning. Otherwise, you can use environment variables with TemplatedConfigLoader, so that A's filename can use yesterday's timestamp, and B today's. So that you also keep a history of your datasets, in case something goes wrong. If something goes wrong however and you don't notice for a few days, you would have to revert and lose all those days' data. You could also combine both approaches if the overhead is too big, and start your aggregation node with say last months dataset as a base and only add this month's days... In case your dataset is not append only and would scale writes with days. In this case, even if something catastrophic happens you still have all the day data backed up, so that you can reconstruct your dataset given enough time. This parallels WAL, which you might find insightful https://en.wikipedia.org/wiki/Write-ahead_logging

How to change the kedro configuration environment in jupyter notebook? https://stackoverflow.com/questions/73984069/how-to-change-the-kedro-configuration-environment-in-jupyter-notebook

Is there a way to include an Azure Databricks Lakehouse query as a DataCatalog dataset in kedro? https://stackoverflow.com/questions/73988370/is-there-a-way-to-include-an-azure-databricks-lakehouse-query-as-a-datacatalog-d

import fsspec throws error (AttributeError: 'EntryPoints' object has no attribute 'get') https://stackoverflow.com/questions/73990243/import-fsspec-throws-error-attributeerror-entrypoints-object-has-no-attribut

Kedro on Databricks: Cannot import SparkDataset https://stackoverflow.com/questions/73999551/kedro-on-databricks-cannot-import-sparkdataset

Let's say I have a Kedro project named

kpro

and I've abstracted a bunch of stuff into

kpro.coollib

. If I package it up with

kedro package

and distribute it, do I just need to do

import kpro.coollib

on my destination machine to use it? In this case I'm not interested in running my pipelines, just the library code. Thanks!

Yes, that would work out of the box. But if you are not interested to ship the pipeline code, you may want to exclude that by modify the setup.py, if you need more details check out the standard python doc about packaging library.

Is anyone having problems with adlfs + partitioned dataset + parallel runner? Apparently, the dataset can't retrieve partitions from blob storage when using this combination. In my tests, it might be something related to asyncio calls inside adlfs glob function.

python
# pipeline_registry.py
pipelines["partitioned"] = Pipeline([node(print, 'partitioned', None)])

yml
# catalog.yml
partitioned:
  type: PartitionedDataSet
  dataset: pandas.CSVDataSet
  path: abfs://...dfs.core.windows/...
  credentials: lab
  filename_suffix: .csv

log
[10/10/22 14:39:54] INFO     Kedro project
[10/10/22 14:39:55] INFO     Loading data from 'partitioned' (PartitionedDataSet)...

I don't have any experience with this, but can you describe what problems did u encounter? What's the error?

The dataset gets stuck when reading data from blob storage

it doesn't happen when saving, but when loading, it gets stuck

I tried to debug kedro in order to find what method it stops, and it stops on

PartitionedDataSet._list_partitions

. it doesn't look like a Kedro problem, but an adlfs problem, since it only stops execution when reaching its glob method.

I tried it in my Windows machine twice, and it worked. For some reason, this is only occuring in Linux

Hello, I have seen in 0.18.3 that

kedro build-docs

etc. will be deprecated in 0.19. Are there other commands which are planned to replace them? Or a new procedure to build the doc or requirements?

Hi @Onéira , the commands will not be replaced, but we will give guidance on alternative workflows. For

kedro build-docs

specifically we have a follow up task to provide a good alternative: https://github.com/kedro-org/kedro/issues/1618 Another thought we have for the commands is to move them to a community supported plugin. Please leave your thoughts on https://github.com/kedro-org/kedro/issues/1622 if this sounds useful to you 🙂

Thanks Merely for your feedback I will check the document you have mentioned ☺️

I have been introducing my team to kedro and multiple questions were raised. I hope you can help me answer these: 1- When

kedro run

in a local machine and it starts executing it says

running kedro session start

does this mean it runs a local server? 2- If we are running Kedro DAG will that mean the kedro will start a local server in our airflow? what ports will it use? 3- The kedro-airflow has been archived, what does this mean for the support of the project? 4- Since there is limited documentation on running kedro in stand-alone airflow, does this mean kedro airflow will only be a sustainable approach with astronomer going forward? 5- why there is kedro operator instead of python or base operator?

Hi @farisology I'll try my best to answer your questions: 1. No Kedro does not use a server to run, unless you set it up that way yourself. Out of curiosity: what version of Kedro are you using? That log line is not in our latest version, and I'm not sure it ever was.. perhaps your team added this for debugging? 2. Are you using the

kedro-airflow

plugin for this? 3. You can definitely run Kedro on stand-alone airflow as well. The Kedro maintainer team tries to keep documentation and deployment guides up to date, but we don't always manage to write about all possible ways of deploying Kedro projects. Let us know where we can help here! 4. I don't really understand your question, is this related to airflow/`kedro-airflow` again?

Pattern wise, does a

GenericDataSet

class factory make sense? I'm looking to write a class factory with the following signature:

python
def create_generic_dataset(
    name: str,
    load: Callable[[Path, ...], pandas.DataFrame],
    save: Callable[[pandas.DataFrame, Path, ...], None]
):
    ...

or is subclassing the

AbstractaDataSet

still the preferred method?

Is this just for pandas? We already have a pandas.GenericDataSet