https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • t

    Thiago Poletto

    08/02/2022, 9:58 AM
    Thank you @noklam
  • b

    brewski

    08/02/2022, 7:41 PM
    Is there a recommended way to reuse nodes/utility functions between different pipelines? micropackaging feels a little overkill for just 3 or so functions
  • b

    brewski

    08/02/2022, 7:51 PM
    also 2nd question: is there a way to directly call a function that defines a node from a jupyter lab session?
  • b

    brewski

    08/02/2022, 7:59 PM
    lol for the 2nd question I just did
    from {{project_name}}.pipelines.{{pipeline_name}}.nodes import *
    and it seemed to work well
  • n

    noklam

    08/02/2022, 9:29 PM
    What stop you from reusing a node between different pipelines?
  • b

    brewski

    08/02/2022, 9:29 PM
    namespaces?
  • n

    noklam

    08/02/2022, 9:30 PM
    Do u want to reuse the function? Or the node
  • b

    brewski

    08/02/2022, 9:30 PM
    function
  • n

    noklam

    08/02/2022, 9:30 PM
    Then it makes sense to just factor it out from the pipeline
  • n

    noklam

    08/02/2022, 9:31 PM
    It could be just a separate python module
  • n

    noklam

    08/02/2022, 9:31 PM
    Same for utils
  • n

    noklam

    08/02/2022, 9:31 PM
    Project.common.xxx
  • n

    noklam

    08/02/2022, 9:32 PM
    Then in your pipeline or notebook u just import from the module instead
  • b

    brewski

    08/02/2022, 9:32 PM
    is the upper level project accessible from the nodes.py file?
  • n

    noklam

    08/02/2022, 9:33 PM
    It's just python code, nothing kedro specific
  • n

    noklam

    08/02/2022, 9:33 PM
    The only specific thing is the namespace structure, but it's not relevant to this
  • b

    brewski

    08/02/2022, 9:38 PM
    very cool
  • b

    brewski

    08/02/2022, 9:38 PM
    didn't know that was a python thing
  • b

    brewski

    08/02/2022, 9:39 PM
    thought you could only see the files in the local folder, unless subdirectories had
    __init__.py
  • n

    noklam

    08/03/2022, 9:21 AM
    This is still valid, by default when you create a new kedro project, let's say the project is called
    demo
    , then you will have
    demo/__init__.py
    ,
    demo/pipelines/__init__.py
    , this is why you can import from there. If you want to factor out some code into a separate module. You will create a new folder and file like
    demo/common/__init__.py
    etc.
  • b

    Binky

    08/04/2022, 3:48 AM
    Hi all. I have a beginner question, on Kedro 0.18.2. I have a TemplatedConfigLoader that loads a partitioned dataset from a specific path
    raw_dataset: 
      type: PartitionedDataSet
      dataset:
        type: pandas.CSVDataSet
        load_args:
          index_col: False
      path: data/01_raw/${asset}
      filename_suffix: ".csv"
      layer: raw
    and my settings.py reads
    CONFIG_LOADER_CLASS = TemplatedConfigLoader
    CONFIG_LOADER_ARGS = {
        "globals_pattern": "*parameters.yml",
    }
    When setup this way my pipeline works as expected, pulling the asset variable from parameters.yml. However now I want to be able to override this variable from the command line e.g.
    kedro run --params=asset:1234
    Any advice on how I can do that? I went on a wild goose hunt on Google and the Kedro github but still can't quite figure it out. It's pretty similar to this issue https://github.com/kedro-org/kedro/issues/1527#issuecomment-1127697871 but i couldn't get the suggested solution to work.
    n
    a
    • 3
    • 15
  • d

    datajoely

    08/04/2022, 7:26 AM
    You need to implement the
    MyTemplatedConfigLoader
    example half way down the page. The CLI arguments will be in
    extra_params
    and you need to ensure that these are passed to
    globals_dict
  • b

    Binky

    08/04/2022, 7:54 AM
    HI @datajoely, I did see and attempt to implement that example, but am still stuck. I access
    extra_params
    by loading the context. I loaded it this way:
    with KedroSession.create("elevator-eventmasker") as session:
        context = session.load_context()
    context.params
    gives the variables in parameters.yml as expected. However
    context._extra_params
    just gives
    None
    rather than the CLI arguments.
  • d

    datajoely

    08/04/2022, 9:29 AM
    They're an argument to the create method
  • n

    noklam

    08/04/2022, 12:45 PM
    Runtime Config to override Catalog
  • t

    Thiago Poletto

    08/04/2022, 1:43 PM
    Hey guys, is there any way to show all features that could be used on Data Catalog, like that autocompletion from docstrings and such when you like, type a method and shows all things you can define within, but for data catalog?
  • t

    Thiago Poletto

    08/04/2022, 1:45 PM
    message has been deleted
  • t

    Thiago Poletto

    08/04/2022, 1:45 PM
    like these ones...
  • t

    Thiago Poletto

    08/04/2022, 1:46 PM
    hey @datajoely how you doing?
  • d

    datajoely

    08/04/2022, 1:46 PM
    In your yaml editor?
Powered by Linen
Title
d

datajoely

08/04/2022, 1:46 PM
In your yaml editor?
View count: 1