https://kedro.org/ logo
#advanced-need-help
Title
# advanced-need-help
f

FelicioV

03/02/2022, 5:42 PM
Hello, I'm trying to implement a few pipelines in Kedro 17.7 that have a lot of inputs of moderate complexity. It can be summarised roughly in reading a few dozen sheets in a few hundred excel spreadsheets. To do so, I'm using
PartitionedDataSets
with
pandas.ExcelDataSet
and specifying
load_args
as
sheet_name
,
names
and
dtype
. It works like a charm but I'm worrying about the size of the
catalog/ingest.yml
. I've been searching for a way to split that catalog yml into a few files, maybe on business oriented segments, but I have had no luck with it. Is there a intended way to do such a thing? If not intended way implemented, I've been thinking (not really tried though) to mess up with the
register_catalog
on the
ProjectHooks
class. Am I making any sense? Thanks!
d

datajoely

03/02/2022, 6:08 PM
Hi @FelicioV this is very doable
I'm not at my computer at the moment but will come back to this later if that's okay!
f

FelicioV

03/02/2022, 6:09 PM
That's perfect. Thanks for your time.
d

datajoely

03/02/2022, 8:03 PM
Hi @User - so you are able to split the catalog into many files
if you look at this sample project that's what I've done here https://github.com/datajoely/modular-spaceflights/tree/main/conf/base
we actually look for the glob path
catalog*
and
catalog*/**
so we will pick up any files that are prefixed with
catalog
or live (recursively) within a folder with that prefix
so you can split into as many files as you want and they're treated as one at runtime
there are also a couple of methods we use to simplify complexity - but they do have tradeoffs in terms of readability
(1) You can use YAML anchors to re-use fragments within the same file
(2) you can use TemplatedConfigLoader to stop repeating yourself
(3) You can use Jinja to use loops in YAML (I'm not a fan of this, because I think it damages readability and makes things hard to maintain, but it's there)
f

FelicioV

03/02/2022, 8:08 PM
Awesome! I'll dig into it. I've tried to nest it once on a folder, got the idea from this snippet on the 01_data_catalog page on the docs
Copy code
# <conf_root>/<env>/catalog/<pipeline_name>.yml
rockets:
  type: MemoryDataSet
scooters:
  type: MemoryDataSet
d

datajoely

03/02/2022, 8:10 PM
I tend to split things into phase-specific areas
but also keep in mind that sometimes that's a premature optimisation
during development I tend to persist everything at every stage so I can pick up where I left off
but once I'm done I delete all the catalog entries in the middle
so you often only need the very beginning inputs and the very end
f

FelicioV

03/02/2022, 8:12 PM
Most of my catalog can, and will, be configured like MemoryDataSets. I'll just persist some cloud steps. Internet connection isn't as reliable as I wish in my area.
d

datajoely

03/02/2022, 8:13 PM
so you don' need to declare memory datasets
if they are outputs to a node the names are addressable in your pipeline logic
and can be referenced at runtime, but don't exist after
f

FelicioV

03/02/2022, 8:13 PM
Got this tip from you on the linkedin live last week
d

datajoely

03/02/2022, 8:14 PM
💪
f

FelicioV

03/02/2022, 8:15 PM
I understand. I'll do it for now to have more examples and later on the implementation I'll comment out most of it. I hope it will be an instructive moment for the team. Again, thanks for your time.
d

datajoely

03/02/2022, 8:15 PM
No worries - happy to review any code snippets if helpful
2 Views