it's not 'kedrific' because we like thinks to be readable at rest, but it is very possible

essentially you would define a hook that retrieved parameters and dynamically added datasets using

catalog.add()

Thanks for the hint. Same applies for Nodes I imagine ?

I mean the preprocessing is the same so in my pipeline I could also have a lifecyle hook ?

or Do I need to add N nodes to the pipelines where N is the number of datasets names ?

Let me try and dig up an example

in the future we want to support the problem you're dealing with natively

as it's a super reasonable use-case

AfterCatalogCreated hook to prepopulate datasets from parameters

Hi all, first of all thanks for the great work done, on kedro! Just had an easy question about the logger. How do I start logging with the journal logger not just the info.log logger?

Hi @User the journal is about to be deprecated so I'd be a little careful about using that one. 1) There are a bunch of settings you can do in

logging.yml

to configure what the various loggers do. 2) In python world you can grab a logger and start using it via

logging.getLogger(name)

I can't exactly remember what the

name

of the the journal logger is, maybe

kedro.journal

but not 100% sure

Cool so I might just add another log then. I guess that would require adding another handler. Not sure how I will allocate the name of the log though?

handlers:
    console:
        class: logging.StreamHandler
        level: INFO
        formatter: simple
        stream: ext://sys.stdout

    info_file_handler:
        class: logging.handlers.RotatingFileHandler
        level: INFO
        formatter: simple
        filename: logs/info.log
        maxBytes: 10485760 # 10MB
        backupCount: 20
        encoding: utf8
        delay: True

    error_file_handler:
        class: logging.handlers.RotatingFileHandler
        level: ERROR
        formatter: simple
        filename: logs/errors.log
        maxBytes: 10485760 # 10MB
        backupCount: 20
        encoding: utf8
        delay: True

    journal_file_handler:
        class: kedro.versioning.journal.JournalFileHandler
        level: INFO
        base_dir: logs/journals
        formatter: json_formatter

we actually have docs here https://kedro.readthedocs.io/en/stable/08_logging/01_logging.html

to introduce a new logger just use the 'key' under handlers

thanks yea I had been reading that but didn't mention the bit about keys. Cool, so logging.getLogger("info_file_handler") is how it would work for the logs/info.log log.

👍

yeah we could probably flesh out the docs a bit more (or if you want to make an open source contribution, feel free to raise a PR 😄 )

Oh yea one final question was curious if other variables are defined for the formatter and how they are defined? Looks like jinja templating I guess.

formatters:
    simple:
        format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

This is actually pure python

we then just take this yaml file and pass it to

logging.config.dictConfig

cool nice never knew about that!

Back with more questions 😃 . Was wondering what the recommended way of testing nodes would be? Particularly if you need some intermediate processing as an input for one of nodes. Do you know if there would be a way to pause runs, then run tests? I guess you could do

kedro run --to-nodes=<node name>

, and go step by step.

The other idea I had was doing a diff on a log. Probably going step by step and doing unit-tests might be a bit more thorough.

So the two ways I do it: 1. Using the CLI like you suggest and doing things like

kedro run --node

etc. 2. Using a proper debugger in VS Code or PyCharm

option 2 is probably better, if you have access to PyCharm you can do amazing things like inspect DataFrames visually

3. Consider data testing like Pandera or Great Expectations so that you can enforce things like schema, cardinality, missing values etc

thanks! yea I am using Great Expectations on the data side. I was also aiming to write a test suite more than anything. So, I was thinking of running things like this and then having a battery of unit tests for each node.

with KedroSession.create("library") as session:
    context = session.load_context()
    print(context.params)
    session.run(<node1 options>)
    run.node1_tests()
    session.run(<node2 options>)
    ... so on

Does that seem reasonable?

It does - but I have an alternative I can dig up