https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • a

    arum

    07/15/2022, 8:01 AM
    I have now tried using astronomer to get kedro project into airflow following this https://kedro.readthedocs.io/en/stable/deployment/airflow_astronomer.html#deployment-process
  • a

    arum

    07/15/2022, 8:02 AM
    this is how it looks now in airflow, kedro module not found
  • d

    datajoely

    07/15/2022, 8:02 AM
    This means you haven't installed requirements on the server
  • a

    arum

    07/15/2022, 8:03 AM
    Well!! i have airflow, kedro and astro installed locally and im doing all of this locally for testing
  • a

    arum

    07/15/2022, 8:03 AM
    so which server would u mean
  • d

    datajoely

    07/15/2022, 8:03 AM
    Or at least the virtual environment that airflow is looking for
    a
    • 2
    • 18
  • a

    arum

    07/15/2022, 2:09 PM
    % which airflow
    /Users/sl/mambaforge-pypy3/envs/airflow_env/bin/airflow
    (airflow_env) % which python 
    /Users/sl/mambaforge-pypy3/envs/airflow_env/bin/python
  • a

    arum

    07/15/2022, 2:13 PM
    and the conf/ is at
    /Users/sl/airflow/
    where the dags folder in airflow.cfg is defined
    dags_folder = /Users/sl/airflow/dags
    and i still have the issue
    raise ValueError(ValueError: Given configuration path either does not exist or is not a valid directory: /conf/base
    . i have read the docs and tried several thing but couldn't succeed. Would be glad if i can get some help here
  • d

    Deep

    07/18/2022, 9:13 AM
    Hi @datajoely, how are you? Just wanted to ask if you have any resource/references which can help me in leveraging kedro functionalities inside Databricks multi node cluster.
  • d

    datajoely

    07/18/2022, 9:46 AM
    So we're working on this as we speak. For now the guide here: https://kedro.readthedocs.io/en/stable/deployment/databricks.html
  • d

    datajoely

    07/18/2022, 9:46 AM
    and some of the mentions in the release notes: https://github.com/kedro-org/kedro/blob/main/RELEASE.md
  • d

    datajoely

    07/18/2022, 9:47 AM
    are the best bet, but if you have anything specific we can assist
  • d

    Deep

    07/18/2022, 9:48 AM
    For now I'm just doing my research. Nothing specific at the moment. Will let you know. Thanks a ton 😁
  • d

    datajoely

    07/18/2022, 9:49 AM
    No worries! Expect it to get better in the next release too
  • d

    Deep

    07/18/2022, 9:50 AM
    Sure thing.
  • a

    arum

    07/18/2022, 10:14 AM
    @datajoely should we wait to hear if any debugging is going on at ur end to suggest any fix? or is the issue still sound to be specific for us?
  • d

    datajoely

    07/18/2022, 10:15 AM
    This isn't an issue - configuration isn't packaged by design you need to copy it over yourself
  • d

    datajoely

    07/18/2022, 10:15 AM
    We follow principles of the 12 factor app which mandates this
  • a

    arum

    07/18/2022, 10:17 AM
    i tried to describe it in git, where we copied the conf/ folder and files ...yet it persists...wonder what would be right path for them to be copied?
  • a

    avan-sh

    07/19/2022, 5:29 PM
    Hi @Deep , been working on databricks setup. Happy to help if you have any issues
  • d

    Deep

    07/19/2022, 5:30 PM
    Hi @avan-sh thats very kind of you buddy. Sending you a fr
  • a

    arum

    07/20/2022, 12:03 PM
    Did anyone here setup airflow instance using kedro-package pipeline? unable to configure, spent good amount of time following the docs, would be great if anyone walked through the same path can assist. The issue is summarised here https://github.com/kedro-org/kedro-plugins/issues/46
    f
    • 2
    • 26
  • f

    Flow

    07/20/2022, 3:09 PM
    Kedro-airflow issue cannot find configu...
  • u

    user

    07/22/2022, 1:47 PM
    Kedro template configuration does not load globals.yml configuration into catalog.yml https://stackoverflow.com/questions/73081209/kedro-template-configuration-does-not-load-globals-yml-configuration-into-catalo
  • b

    brendalf

    07/23/2022, 10:05 PM
    Hi there. I created a custom dataset to manage some spark delta tables. One thing I need in order to save the data is to receive the replace where query. Is there any way to send this info plus the data from the previous node execution or recover this information inside the custom dataset? The reason why I can’t use the parameters file in the usual way is because I need values defined in execution time
  • u

    user

    07/25/2022, 7:39 AM
    Kedro template configuration does not load/parse variables https://stackoverflow.com/questions/73105524/kedro-template-configuration-does-not-load-parse-variables
  • b

    brendalf

    07/25/2022, 1:14 PM
    https://github.com/kedro-org/kedro/issues/1723
  • d

    Deep

    07/26/2022, 10:27 AM
    Hi @avan-sh, been trying to reach out to you. Please dm me as I have some queries. Take your time, no hurry.
  • n

    Nick Sieraad

    07/28/2022, 7:43 AM
    Hi all, I am using Kedro v0.18.2 and I have a question. When I want to config the catalog I got an error (Error.png). The error is in the globals.yml (globals yml.png) where I specified the dataset that is initialised in my folder 'base' (__init__ from base). When the file, where I config the catalog, is a folder level above 'base' it works, but when the file (nodes.py) is on a lower level (Folder structure) it doesn't work. I tried to specify the whole path in the globals.yml, but that doesn't work. I hope you can help me out!
    n
    • 2
    • 16
  • n

    noklam

    07/28/2022, 10:25 AM
    @Nick Sieraad > When the file, where I config the catalog, is a folder level above 'base' it works, but when the file (test.py) is on a lower level (Folder structure) it doesn't work. I tried to specify the whole path in the globals.yml, but that doesn't work. I hope you can help me out! Which file are you talking about? Is this coming from the StackOverFlow question?
Powered by Linen
Title
n

noklam

07/28/2022, 10:25 AM
@Nick Sieraad > When the file, where I config the catalog, is a folder level above 'base' it works, but when the file (test.py) is on a lower level (Folder structure) it doesn't work. I tried to specify the whole path in the globals.yml, but that doesn't work. I hope you can help me out! Which file are you talking about? Is this coming from the StackOverFlow question?
View count: 2