So this isn't an area that Kedro excels at. In general we only have 1 decent way of doing remote execution on a SQL database and that's via Spark and it's predicate pushdown features. This isn't ideal in all cases because it adds overhead, but it's the most pythonic way of doing things. 1. Unfortunately happens via our pandas datasets which is sub-optimal for big datasets 2. Feels like a better solution - perhaps it's even a chance to use dbt for the munging processes and Kedro for the parts that need to live in python

Hi, if I try to specify a custom separator (tabs, so

'\t'

) for a

Pandas.CSVDataSet

I get a

"delimiter" must be a 1-character string

error. Is there another way to accomplish this?

Can you post your yaml?

Can you try removing the quotes from the sep argument?

It still fails with the same error

Okay worked it out - it's a YAML escaping thing:

yaml
sep: "\t"

Awesome, thanks!

This may be a silly question, but when writing a test suite how do you go about generating input data for said tests? My current engagement is dealing with a large dataset that takes a couple minutes to load into memory so using the actual data isn't a good option. Are there any good examples of projects that do this well when the source data is too large?

@User For what sort of tests? If we're talking unit tests, it makes sense to hand-craft the input data. On the other hand, for end-to-end pipeline tests, consider using a data mocker to generate fake data (if you have stuff like data security restrictions), or simply subsampling your source data. Note that creating test data can be a pretty big challenge in and of itself, depending on how many tables you need to mock and how realistic it needs to be.

I would also note there are few times that enormous datasets in tests tell you anything about the correctness of your code.

A pipeline returns a dictionary of dataframes where the key is the identifier/name of the value(pandas dataframe). I wish to write the dataframes to CSV. How should I go about it?

Hello are you using the kedro code api? not the project template that exposes the YAML api and allows you to save catalog entries

I was looking for a way where I can specify the output in a catalog.yml

Ah okay - so we have lots of examples in the docs but essentially you need to put a catalog entry with the key you want to save

I’ll post the catalog docs, but I’d highly recommend following the tutorial project as this is covered

https://kedro.readthedocs.io/en/stable/05_data/01_data_catalog.html https://kedro.readthedocs.io/en/stable/03_tutorial/01_spaceflights_tutorial.html

I'm in the process of going through the tutorials. Thanks!

Good luck!

To start a conversation about credentials management - global file or cloud native support - would it be best to do it here or keep it localized to the github issues I just opened?

let's keep it on github- it's a really good topic to think through as we're going through a wider configuration design effort

flipping love that Waylon is on here as an emoji!

@User is the 👑 king kedroid 🙂

we work together so I gotta get my stats up publicly I guess!

Laughing so hard at this!

Is there some way to “run” a packaged pipeline from within another python project

I can “python -m my_pipeline” fine, I’m wondering if I can “import my_pipeline” and then “my_pipeline.run” or something similar

So sort of - you can use the code API and do something like:

python
from some_package import kedro_pipeline

from kedro.io import DataCatalog
from kedro.runner import SequentialRunner

catalog = DataCatalog(data_sets=datasets, feed_dict={"params:split_args": params})
output = runner.run(pipe, catalog)

but also maybe

micropackaging

is what you you are looking for https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/04_micro_packaging.html

Anybody know if if it's possible to auto-sign commits from the GitHub UI? Super annoying to have to go back and rebase just because I forgot to manually add it when applying a suggestion from code review or something.