The

register_config_loader

hook (https://kedro.readthedocs.io/en/latest/kedro.framework.hooks.specs.RegistrationSpecs.html#kedro.framework.hooks.specs.RegistrationSpecs.register_config_loader) also gets all the extra_params you'll be passing from command line. You can use that to add file_path to globals_dict. And also your filepath placeholder will be

${filepath}

. LMK if I'm not clear, I can help get a quick snippet later if you want.

@User yeah this is possible by tweaking the config loader like @User suggests - you also make env variables available as $vars this way: https://discord.com/channels/778216384475693066/846330075535769601/950324995752620052

A snippet would help to understand that approach. I'm not sure how to modify the hook. I just tried @User suggestion, and that works nicely so I got my question answered, thanks!

An approach like this should work as well. One extra note, your runs might fail if you don't have any default value even when you're not using those items.

class ProjectHooks:
    @hook_impl
    def register_config_loader(conf_paths, env, extra_params) -> ConfigLoader:
        globals_dict = {}
        globals_dict["filepath"] = extra_params.get("filepath", "default_value")
        return TemplatedConfigLoader(
            conf_paths,
            globals_dict=globals_dict,
        )

I'll give that alt a go too.

Why i'm getting access denied?

yaml
# catalog.yml
example_data:
  type: PartitionedDataSet
  dataset: pandas.ParquetDataSet
  credentials: dev_s3
  path: s3://bucket/path/to/folder
  filename_suffix: "_part.parquet"

yaml
# credentials.yml
dev_s3:
  client_kwargs:
    aws_access_key_id: xxxx
    aws_secret_access_key: xxx

If it's coming from boto3 underneath the error then it's credentials not kedro

I was using the same credential to read, but notice that it doesnt have permissions to write 😅

Guys, how are you all? I've been using kedro for the past months and I've really been enjoying it 😄

Quick question. Is there a way of logging dataset sizes elegantly? Rather than a per node writting a logging.info statement?

This is the right time to use hooks! It would be very similar to this Memory profiling examples: https://kedro.readthedocs.io/en/latest/07_extend_kedro/02_hooks.html#hooks-examples

Can you reference parameters in a list by their index? For example if I had a file

parameters.yml

with the following contents:

yaml
folds:
  timeseries1:
    - start: 2021-08-24 15:00:00+00:00
      end: 2021-10-22 03:05:00+00:00
    - start: 2021-10-22 03:10:00+00:00
      end: 2021-12-28 05:00:00+00:00     
    - start: 2021-12-28 05:05:00+00:00
      end: 2022-01-28 12:00:00+00:00

Can I reference the second [start, end]? I've tried using something like

params:folds.timeseries1.1

or

params:folds.timeseries1[1]

, but that doesn't work.

Unfortunately not - you would have to either do the indexing in Python or if you really wanted to make the keys the index yourself

Ok, thanks for the extremely quick reply. I'll simply use the second method.

Hello, I was just wondering if it's normal to kedro executes hooks.py twice during

kedro run

It is not! IIRC there is an old version where this happened, what version are you running?

'0.17.7'

so that's unexpected - are you using custom hooks? What logging message to you see?

I just print the conf paths and logs show it twice... I'll check the plugins and reinstall everything

aah yes, this seems the exact use case. But all the examples only work with the DataSet name. Is the recommended approach to have the catalog loaded, read it and get the dimmensions?

The nodes shouldn't take any responsibility for loading, if you find yourself creating a catalog within a node you've gone too far! Nodes should be pure in the functional sense. The hooks let you touch the catalog in the right way.

not within the node itself, but within the hook I'd like to log the dimensions of the data!

Then that's exactly what they're for

Hi Team, What is the best way to get Run ID from node code?

Hooks!

are you saying run_params of after_pipeline_run()? can I access run_params from node code?

So you can look at the specs for all of our hooks here https://kedro.readthedocs.io/en/stable/kedro.framework.hooks.specs.html

lets' say my code likes this ##node_07_evaluate_model.py def evaluate_model(model, X_train, y_train, test_set, params, globals_): plot_writer = MatplotlibWriter( filepath="data/08_reporting/test_model_plot.png", version=Version(load=None, save=None) )

i'd like to get run_id to save

test_model_plot.png

under the same version(run_id)

The node shouldn't have knowledge of how things get saved