https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • g

    gonecaving

    10/17/2022, 7:32 AM
    @noklam yes that’s exactly what I meant. I get that it adds a significant layer of complexity, however I’m not sure that I see a better way of enabling users to construct a pipeline from a pre-defined set of building blocks without writing code. Thanks for the pointer towards the hooks, I’ll give that a look.
  • u

    user

    10/17/2022, 3:09 PM
    Include Quarto rendering in kedro pipeline and pass it inputs/outputs https://stackoverflow.com/questions/74099238/include-quarto-rendering-in-kedro-pipeline-and-pass-it-inputs-outputs
  • t

    ToniMaroni

    10/18/2022, 7:57 AM
    Hi, is there a way to have the configurations in settings.py configurable based on the kedro environment. For example, I want the hooks for kedro-viz and kedro-neptune disabled in production environment. Thanks
  • d

    datajoely

    10/18/2022, 7:59 AM
    Interesting usecase - best solution off the top of my head is to use the KEDRO_ENV environment variable and just include an if statement in settings.py
  • t

    ToniMaroni

    10/18/2022, 8:00 AM
    I was also thinking of a using an ENV var, glad to hear that there is one already defined for the KEDRO_ENV! Thanks
  • d

    datajoely

    10/18/2022, 8:01 AM
    The other option, which is probably preferable, is to do the logic in the hooks themselves which do have the env name as a parameter
  • d

    datajoely

    10/18/2022, 8:01 AM
    This isn't possible if you're using 3rd party hooks
  • n

    noklam

    10/18/2022, 8:02 AM
    https://github.com/kedro-org/kedro/issues/1935
  • n

    noklam

    10/18/2022, 8:02 AM
    Some similar requests a few days ago
  • d

    datajoely

    10/18/2022, 8:02 AM
    Nice one @noklam !
  • n

    noklam

    10/18/2022, 8:03 AM
    https://github.com/kedro-org/kedro/issues/1842
  • t

    ToniMaroni

    10/18/2022, 8:20 AM
    the KEDRO_ENV does not seem to be set by the time the session creation does the "validate_settings()"
  • d

    datajoely

    10/18/2022, 8:21 AM
    Sorry it doesn't get set automatically (it may in the future) but you can use it to set the env pre run
  • d

    datajoely

    10/18/2022, 8:21 AM
    So it's instead of the CLI arg
  • t

    ToniMaroni

    10/18/2022, 8:21 AM
    ok thanks!
  • e

    eliaspd

    10/18/2022, 12:32 PM
    Hey, I was wondering if there was an elegant solution to overwrite parameters dynamically? I am instanciating a pipeline 12 times, but they all need to run with a different parameter called date_max, e.g. "07/01/22" for the first one, and the other ones are decrementing one month, e.g. "06/01/22"... etc.. The pipelines are generated of a template dynamically and ideally I would just pass the adjusted parameter.
  • e

    eliaspd

    10/18/2022, 12:34 PM
    parameters.yml
    t_-0:
      filters:
        date_max: 2022/07/01
    t_-1:
      filters:
        date_max: 2022/06/01
    So I want to avoid doing this, as I would need to pass 12 or more variables on each induction. Whereas they are actually all are dependent on the first one.
  • d

    datajoely

    10/18/2022, 2:00 PM
    Hooks!
  • d

    datajoely

    10/18/2022, 2:00 PM
    Although we've talked a lot about the dangers of dynamic pipelines in terms of complexity and maintenance overhead
  • e

    eliaspd

    10/18/2022, 2:03 PM
    Could you elaborate a little the hooks solution?
  • d

    datajoely

    10/18/2022, 2:21 PM
    Same concept https://discord.com/channels/778216384475693066/846330075535769601/931130020762648576
  • b

    bluesummers

    10/19/2022, 7:18 AM
    Quick Q, we can create our custom starters, which is very useful - we use it to set company-wide standards. Is there a way to create custom pipeline layouts? So when we do
    kedro pipeline create
    it will use a custom cookiecutter layout created by us? Technically it seems possible as I see in the
    kedro
    repo there is a dedicated cookiecutter for pipelines
    n
    d
    • 3
    • 14
  • u

    user

    10/19/2022, 2:25 PM
    How can I run
    catalog.load
    in a non-IPython context?
    https://stackoverflow.com/questions/74127089/how-can-i-run-catalog-load-in-a-non-ipython-context
  • p

    p3t3r

    10/20/2022, 6:19 AM
    Hi, I am trying to run my kedro project in Apache Airflow. I follow the instructions and am able to launch the Airflow app when I do not attach my kedro image in docker-compose.yaml. However, when I do attach the image, by running docker-compose up, I get: Starting airflow_redis_1 ... done Starting airflow_postgres_1 ... done Starting airflow_airflow-init_1 ... done Starting airflow_airflow-webserver_1 ... done Starting airflow_airflow-triggerer_1 ... done Recreating airflow_airflow-worker_1 ... error Recreating airflow_airflow-scheduler_1 ... Recreating airflow_airflow-scheduler_1 ... error ERROR: for airflow_airflow-scheduler_1 Cannot start service airflow-scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "scheduler": executable file not found in $PATH: unknown ERROR: for airflow-worker Cannot start service airflow-worker: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "celery": executable file not found in $PATH: unknown ERROR: for airflow-scheduler Cannot start service airflow-scheduler: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "scheduler": executable file not found in $PATH: unknown ERROR: Encountered errors while bringing up the project.
  • d

    datajoely

    10/20/2022, 6:24 AM
    I don't think any of this is kedro specific - it looks like docker/airflow configuration
  • p

    p3t3r

    10/20/2022, 6:31 AM
    yes, it does seem like that. However, I was wondering if someone has encountered this before and could help me with this. Or if kedro documentation could be improved to avoid such configuration issue?
  • d

    datajoely

    10/20/2022, 6:32 AM
    It's not something I've seen before unfortunately
  • b

    bluesummers

    10/20/2022, 12:26 PM
    Trying to create a Kedro starter integrated with Poetry. I'm following this issue (https://github.com/kedro-org/kedro/issues/1722) to get it done. Looking at the docs in this link (https://kedro.readthedocs.io/en/stable/extend_kedro/plugins.html#extend-starter-aliases) I should edit
    setup.py
    to get it done. Now... (1) I'm creating a starter, so there isn't any
    setup.py
    yet as I have not executed Kedro new. So how exactly am I supposed to edit it if it is not yet created? Second (2), I don't want to use
    setup.py
    , as Poetry does not need it. So how should I go about supporting new kedro starters?
    d
    • 2
    • 24
  • m

    Merel

    10/21/2022, 9:50 AM
    Running Kedro on Airflow
  • u

    user

    10/23/2022, 7:57 AM
    How to show plotly chart in kedro https://stackoverflow.com/questions/74169483/how-to-show-plotly-chart-in-kedro
Powered by Linen
Title
u

user

10/23/2022, 7:57 AM
How to show plotly chart in kedro https://stackoverflow.com/questions/74169483/how-to-show-plotly-chart-in-kedro
View count: 3