https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • d

    datajoely

    02/08/2022, 9:35 PM
    I’m not really sure thought - I don’t use conda outside of venvs
  • e

    erbabydriv

    02/09/2022, 3:23 AM
    Hey All! New Kedro user here.. I'm trying to use Kedro test - is there any option to get detailed test summary and assertion errors? I can only see short test summary
    a
    • 2
    • 2
  • r

    RRoger

    02/09/2022, 4:48 AM
    What's the naming convention in catalogs when using modular pipelines? In https://github.com/datajoely/modular-spaceflights *
    catalog_01_raw.yml
    has "companies", "reviews" (no prefixes) *
    catalog_02_int.yml
    has "data_ingestion.int_typed_companies", "data_ingestion.int_typed_shuttles" (prefix with dot) *
    catalog_03_prm.yml
    has "prm_shuttle_company_reviews", "prm_spine_table" (prefix with underscore) Or does is it a matter of taste?
    c
    d
    a
    • 4
    • 5
  • a

    avan-sh

    02/09/2022, 5:07 AM
    kedro-test
  • c

    ChainYo

    02/09/2022, 5:46 AM
    Data Catalog
  • g

    ggerog

    02/09/2022, 4:16 PM
    Hi all, I was wondering if anyone knows how I can grab the kedro env flag from within a pipeline.py file, if I run
    kedro run --env=test
    for instance. I want to pick up the
    test
    string. Thanks,
  • d

    datajoely

    02/09/2022, 4:21 PM
    So two ways - you can use a
    hook
    or if you use the
    KEDRO_ENV
    variable to set it you could just get that within your node
  • g

    ggerog

    02/09/2022, 4:49 PM
    Cool, thanks maybe the environment variable seems like an easier fix!
  • d

    datajoely

    02/09/2022, 4:50 PM
    There is an argument a
    before_node_run
    hook is the best way to do this, but I think the env variable one is neat
  • g

    ggerog

    02/09/2022, 4:52 PM
    > so how would I get the return of
    before_node_run
    into a variable. I mostly just use hooks to get the side-effects.
  • d

    datajoely

    02/09/2022, 4:53 PM
    so you could map the
    inputs
    dict to a new value
  • d

    datajoely

    02/09/2022, 4:54 PM
    but this is more of a philosophical point - we like to think of nodes as pure python functions that have no knowledge of IO operations or execution logic
  • d

    datajoely

    02/09/2022, 4:54 PM
    so if you can do it outside of a node logic it feels more Kedrific
  • g

    ggerog

    02/09/2022, 4:55 PM
    so, the output of
    before_node_run
    , can be used as an input?
  • d

    datajoely

    02/09/2022, 4:56 PM
    you can mutate the
    inputs
    provided to the node
  • g

    ggerog

    02/09/2022, 4:58 PM
    not quite sure how I would mutate the inputs from
    before_node_run
  • d

    datajoely

    02/09/2022, 4:59 PM
    At this point - maybe we should say the env variable is the simplest solution!
  • d

    datajoely

    02/09/2022, 4:59 PM
    I've made a note to document this as it comes up a lot
  • g

    ggerog

    02/09/2022, 5:02 PM
    one last idea, could you not get the hook to point to a memory_dataset and then read that (in this case the flag for env)
  • d

    datajoely

    02/09/2022, 5:17 PM
    Yeah that's what I was getting to!
  • d

    datajoely

    02/09/2022, 5:17 PM
    nicely put
  • g

    ggerog

    02/09/2022, 5:22 PM
    I ended up doing something slightly amusing. Just combined your ideas used the default project hook to setup the KEDRO_ENV programmatically via python haha.
  • d

    datajoely

    02/09/2022, 5:38 PM
    I like it!
  • d

    datajoely

    02/09/2022, 5:38 PM
    at first I was very confused
  • d

    datajoely

    02/09/2022, 5:39 PM
    maybe we should make that default behaviour 🤔
  • g

    ggerog

    02/10/2022, 8:11 AM
    yea I think that could work quite well.
  • i

    Isaac89

    02/10/2022, 3:22 PM
    Hi @User! I was testing this idea but it seams that Hook instances are different when the before_pipeline_run and before_node_run are executed, so the attribute is reset. Is it possible that the Hook class is instantiated twice ? I can see the same attribute in after_catalog_created, before_pipeline_run, but it is None in the berfore_node_run.
  • a

    antony.milne

    02/10/2022, 3:30 PM
    hmmm, that's weird. The class is only instantiated once (in
    settings.py
    ) but possibly there's something else going on that's obliterating the attribute. Let me try it out myself and see if it works
  • a

    antony.milne

    02/10/2022, 3:35 PM
    Can confirm that it works ok for me. I think you must have some additional logic going on somewhere that is overwriting the attribute somehow?
  • d

    datajoely

    02/10/2022, 3:36 PM
    Do you have multiple hooks?
Powered by Linen
Title
d

datajoely

02/10/2022, 3:36 PM
Do you have multiple hooks?
View count: 1