Zhee
11/04/2021, 1:37 PMdatajoely
11/04/2021, 6:57 PMdatajoely
11/05/2021, 10:22 AMkedro-viz
to have some sort of 'catalog manager' where this would make sense to live. It's not under active development, but if users start shouting that they'd like it has more weight on the backlog ๐
- Finally, the most structured way of doing this today is to use Sphinx and the built in kedro build-docs
command to generate static docs. This is mostly there for Python API docs, but everything on the Kedro docs (https://kedro.readthedocs.io/en/stable/) is made this way so you can steal how we do it too. I think it would be pretty neat to write a script that used the DataCatalog python API to create data documentation stubs which you then fill in with human readable descriptions.Zhee
11/05/2021, 1:52 PMdatajoely
11/05/2021, 1:59 PMBarros
11/05/2021, 5:10 PMdatajoely
11/05/2021, 5:10 PMdatajoely
11/05/2021, 5:11 PMdatajoely
11/05/2021, 5:11 PMBarros
11/05/2021, 5:12 PMBarros
11/05/2021, 5:13 PMdatajoely
11/05/2021, 5:13 PMdatajoely
11/05/2021, 5:14 PMBarros
11/05/2021, 5:15 PMBarros
11/05/2021, 5:15 PMBarros
11/05/2021, 5:15 PMZemeio
11/05/2021, 10:59 PMyml
# globals.yml
env:
base:
folder: "data/prod"
test:
folder: "data/test"
folders:
# Base folders, where the main pipelines are run
raw: "${env.base.folder}/01_raw"
int: "${env.base.folder}/01_intermediate"
# Test folders, where smaller subsets of the data designed for testing reside, and the test pipelines run
test_raw: "${env.test.folder}/01_raw"
test_int: "${env.test.folder}/01_intermediate"
However, when I try to run this it does not get the value from env.base. Does the templated config loader only templates for catalog.yml? Is the only official way to do this using jinja templates?datajoely
11/06/2021, 11:51 AMconf/prod
and conf/test
and then get Kedro to resolve which one you want a run time by doing kedro run --env=prod
or use the env variable export KEDRO_ENV=prod
Zemeio
11/06/2021, 1:25 PMZemeio
11/06/2021, 1:25 PMdatajoely
11/08/2021, 9:38 AMMatheus Serpa
11/09/2021, 9:59 PMdatajoely
11/10/2021, 9:47 AMsemente_table
kedro doesn't know which one to write first!datajoely
11/10/2021, 9:48 AMload_data
which is called something like processed_semente_table
datajoely
11/10/2021, 9:48 AMMatheus Serpa
11/10/2021, 1:34 PMdatajoely
11/10/2021, 1:49 PMIsaac89
11/10/2021, 1:51 PMdatajoely
11/10/2021, 1:52 PMdatajoely
11/10/2021, 1:54 PMdatajoely
11/10/2021, 1:54 PM