https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • d

    datajoely

    09/06/2021, 1:49 PM
    Okay thank you @User let me ask one of the devs - thanks for checking
  • e

    ende

    09/06/2021, 6:53 PM
    Thanks @User - How about within a monorepo.. would you say it makes sense to have separate projects within a monorepo, or pile all the pipelines together into a single project?
  • d

    datajoely

    09/07/2021, 8:00 AM
    I really it think it depends on how much logic/shared catalog exists between the pipelines. I think I would err on one repo per Kedro project, but utilise multiple pipelines and configuration environments within that project.
    w
    e
    • 3
    • 4
  • d

    datajoely

    09/07/2021, 8:03 AM
    Which version of Kedro are you working with?
  • d

    datajoely

    09/07/2021, 8:46 AM
    There were fixes to this dataset in 0.17.4 and 0.17.1
  • i

    Isaac89

    09/07/2021, 9:19 AM
    Hi everyone! Is there any way to generate the catalog config from a DataCatalog object. Something like DataCatalog.to_config() ? Has anyone already done something like this ? Thanks in advance for your help!
  • u

    user

    09/07/2021, 12:25 PM
    Versioned SparkDataSet problem
  • m

    Malaguth

    09/07/2021, 1:47 PM
    I'm working with Kedro 0.17.4
  • e

    ende

    09/07/2021, 8:55 PM
    Perfect. Thanks! Obviously a very context-dependent question, but I just wanted to get a general sense for a good starting point.
  • w

    waylonwalker

    09/08/2021, 3:33 AM
    monorepo?
  • m

    Malaguth

    09/08/2021, 9:20 PM
    Hello everyone. I was thinking about pipeline reuse in kedro and want to know the best way to do that. In my first pipeline, I segment my dataset into subsets dynamically. After that, I need to run a second pipeline for each set. Is it possible to run the same pipeline for each input dynamically? Does anyone already handle dynamic inputs?
  • d

    datajoely

    09/09/2021, 7:33 AM
    So the discussion on dynamic pipelines comes up a lot. Our default position is that readability is the most important thing. You can use Jinja to define dynamic catalogs and regular Python to generate dynamic pipelines. That being said - we've observed that things become very difficult to maintain things when the pipeline logic only really exists at runtime rather than rest. With this in mind, if you do dynamic pipelines I'd encourage you to dynamically generate pipelines that are immutable in structure, but dynamic in terms of inputs/outputs. If you start building conditional logic into your pipeline generation it becomes very difficult for other team members or 'future you' to read in 6 months time
  • d

    datajoely

    09/09/2021, 7:34 AM
    The other thing to keep in mind is custom configuration environments https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#additional-configuration-environments This is where you have mirrors of the same datasets for different environments e.g. staging vs prod
  • w

    WolVez

    09/09/2021, 10:40 PM
    @User it looks like a set of recent releases from dynaconf are causing breaking effects (shown below). It seems stable on dynaconf==3.1.5.
  • d

    datajoely

    09/10/2021, 6:21 AM
    Yes we’ve just got reports of this elsewhere
  • d

    datajoely

    09/10/2021, 6:21 AM
    I’m looking into it
  • d

    datajoely

    09/10/2021, 7:59 AM
    Thanks @User - we'll put a proper fix out shortly, but for now please pinning to 3.1.5 looks like it fixes things!
  • z

    Zhee

    09/10/2021, 2:25 PM
    Had same issue with dynaconf 3.1.7 today. Rollbacked to 3.1.4 it was fine. Will test with 3.1.5.
  • d

    datajoely

    09/10/2021, 2:30 PM
    0.17.5 was imminent annyway and we'll put a fix for this in too
  • w

    WolVez

    09/10/2021, 4:32 PM
    @User do you have an estimated release date for 0.17.5? Also, are there expected to be any breaking changes?
  • d

    datajoely

    09/10/2021, 4:40 PM
    Next week is my hope - we do our absolute to not do any breaking changes in 'minor releases' e.g. 0.17.4->0.17.5
  • d

    datajoely

    09/10/2021, 4:40 PM
    the next breaking changes will happen in 0.18.0 which I'm hoping for Q4 2021
  • d

    datajoely

    09/10/2021, 4:41 PM
    and that will put us on the path to have a really stable API ready for 1.0
  • y

    Yetunde

    09/13/2021, 3:41 PM
    @User It should be out by tomorrow! We also have a new release for Kedro-Viz coming out soon.
  • y

    Yetunde

    09/15/2021, 11:01 AM
    And we have a new Kedro release out! Check the #778996598880862208 channel!
  • s

    ScttOptionz

    09/20/2021, 10:01 PM
    I might be completely stupid, but is it possible to split the catalog.yml file into different yaml files that kedro can integrate with?
  • d

    datajoely

    09/21/2021, 3:20 PM
    Yes
  • d

    datajoely

    09/21/2021, 3:21 PM
    To it being possible not you being stupid!
  • d

    datajoely

    09/21/2021, 3:21 PM
    Essentially we match a pattern
  • d

    datajoely

    09/21/2021, 3:24 PM
    So being the scenes we do this
    conf_loader.get('catalog*', 'catalog*/**')
Powered by Linen
Title
d

datajoely

09/21/2021, 3:24 PM
So being the scenes we do this
conf_loader.get('catalog*', 'catalog*/**')
View count: 1