https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • g

    gui42

    03/17/2022, 5:06 PM
    aah yes, this seems the exact use case. But all the examples only work with the DataSet name. Is the recommended approach to have the catalog loaded, read it and get the dimmensions?
  • d

    datajoely

    03/17/2022, 5:09 PM
    The nodes shouldn't take any responsibility for loading, if you find yourself creating a catalog within a node you've gone too far! Nodes should be pure in the functional sense. The hooks let you touch the catalog in the right way.
  • g

    gui42

    03/17/2022, 5:12 PM
    not within the node itself, but within the hook I'd like to log the dimensions of the data!
  • d

    datajoely

    03/17/2022, 5:13 PM
    Then that's exactly what they're for
  • d

    Daehyun Kim

    03/17/2022, 6:54 PM
    Hi Team, What is the best way to get Run ID from node code?
  • d

    datajoely

    03/17/2022, 6:55 PM
    Hooks!
  • d

    Daehyun Kim

    03/17/2022, 7:00 PM
    are you saying run_params of after_pipeline_run()? can I access run_params from node code?
  • d

    datajoely

    03/17/2022, 7:01 PM
    So you can look at the specs for all of our hooks here https://kedro.readthedocs.io/en/stable/kedro.framework.hooks.specs.html
  • d

    Daehyun Kim

    03/17/2022, 7:01 PM
    lets' say my code likes this ##node_07_evaluate_model.py def evaluate_model(model, X_train, y_train, test_set, params, globals_): plot_writer = MatplotlibWriter( filepath="data/08_reporting/test_model_plot.png", version=Version(load=None, save=None) )
  • d

    Daehyun Kim

    03/17/2022, 7:02 PM
    i'd like to get run_id to save
    test_model_plot.png
    under the same version(run_id)
  • d

    datajoely

    03/17/2022, 7:02 PM
    The node shouldn't have knowledge of how things get saved
  • d

    Daehyun Kim

    03/17/2022, 7:02 PM
    i see
  • d

    Daehyun Kim

    03/17/2022, 7:03 PM
    so we need to use catalog,?
  • d

    datajoely

    03/17/2022, 7:03 PM
    You should get something like that by just adding
    versioned: True
    to you catog entry
  • d

    datajoely

    03/17/2022, 7:03 PM
    We do that already
  • d

    datajoely

    03/17/2022, 7:04 PM
    If you really wanted to control things you could do some hook stuff to mutate the catalog live bit I wouldn't recommend it
  • d

    Daehyun Kim

    03/17/2022, 7:04 PM
    ok thanks
  • d

    datajoely

    03/17/2022, 7:41 PM
    @lbonini https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/04_micro_packaging.html
  • l

    lbonini

    03/17/2022, 7:44 PM
    You answered just when I founded 😅
  • l

    lbonini

    03/17/2022, 7:44 PM
    Thanks
  • d

    Daehyun Kim

    03/17/2022, 7:50 PM
    @User is there a catalog data types that handles png file or Matplotlib objects?
  • d

    datajoely

    03/17/2022, 7:51 PM
    https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.matplotlib.MatplotlibWriter.html
  • d

    Daehyun Kim

    03/17/2022, 8:58 PM
    thank you
  • d

    Daehyun Kim

    03/17/2022, 8:58 PM
    it's working great!
  • g

    gui42

    03/18/2022, 12:36 AM
    Guys, I'm in love with hooks 😄
  • g

    gui42

    03/18/2022, 12:36 AM
    Seriously, it is so fun to write them hahaha
  • f

    FelicioV

    03/21/2022, 11:44 AM
    Hello! I've been trying to find examples of credentials for azure blobs with no success. Anyone have an easy snippet to share? Thanks!
  • f

    FelicioV

    03/21/2022, 11:49 AM
    Just occur to me to look on the fsspec implementation, maybe I'l have some luck over there
  • d

    datajoely

    03/21/2022, 11:55 AM
    Hello @User I'm sure we do - what dataset are you trying to use? and which Azure protocol?
  • f

    FelicioV

    03/21/2022, 11:58 AM
    I'm aiming for something able to read a parquet dataset on my azure data blob. No preferences yet, all I have is some clue that the
    abfs://
    protocol with the
    pandas.ParquetDataset
    might be what I'm looking for.
Powered by Linen
Title
f

FelicioV

03/21/2022, 11:58 AM
I'm aiming for something able to read a parquet dataset on my azure data blob. No preferences yet, all I have is some clue that the
abfs://
protocol with the
pandas.ParquetDataset
might be what I'm looking for.
View count: 1