https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • l

    LawrenceS

    08/23/2022, 12:57 PM
    I agree, I think they're useful for those reasons, so would like to commit them to the repository! Do you use any tools, like nbconvert or anything to deal with committing notebooks to avoid merge conflicts?
  • k

    Krab

    08/24/2022, 6:35 AM
    Hi, Question regarding dependencies - would anyone know if fsspec upper boundary is going to be reviewed soon? We are limited to use fsspec >=2022.5.0 so it doesn't play with Kedro 0.18.2. which is limited to <=2022.1.
  • n

    noklam

    08/24/2022, 7:43 AM
    Can you share why you need a relatively new version of fsspec? Would be great if you can open an GitHub issue about this. We generally don't set upperbound unless there is something breaking. I will have a look today.
  • k

    Krab

    08/24/2022, 7:57 AM
    There is a chain of dependencies on our side in fact. We need to use package developed by another team that requires s3fs = ">=2022.5.0", which requires mentioned version of fsspec. I read through the history of the fsspec requirement limitation in Kedro repo and seems that there was some breaking changes in the past hence the cautiousness.
  • k

    Krab

    08/24/2022, 8:29 AM
    Issue opened. Thank you @noklam
  • n

    noklam

    08/24/2022, 8:31 AM
    Thank you
  • v

    Vici

    08/24/2022, 8:43 AM
    The python package "nbstripout" works well for me. Install it, add three lines to your .gitattributes file, and it strips all outputs and metadata from a notebook upon "git add *", which makes for a clean git workflow. https://github.com/kynan/nbstripout
  • t

    Thiago Poletto

    08/24/2022, 1:22 PM
    hey guys, I was wondering if you guys know any tool that could be used with Kedro that works well obviously, that provides the template for docstrings... I knew some, but they are not up to date anymore...
  • d

    datajoely

    08/24/2022, 2:19 PM
    I'm not sure what you mean - I rely on my IDE configuration to define my doc string templates. Pycharm is best at this, Vs code is ok. For validating format there are also lining tools available
  • t

    Thiago Poletto

    08/24/2022, 2:21 PM
    I don't use pycharm, but I mean, there was a tool that was like flake8 that you run it and it does create a docstring template for every function you have in the project...
  • t

    Thiago Poletto

    08/24/2022, 2:21 PM
    so that you wouldn't do it manually
  • d

    datajoely

    08/24/2022, 2:22 PM
    I'm not sure if any tools like that. Flake8 will tell you where they're missing but I don't think it will change you code for you. Googling suggests there is a pycharm function to bulk add missing docstrings
  • t

    Thiago Poletto

    08/24/2022, 2:29 PM
    nice...I'll take a look at that
  • t

    Thiago Poletto

    08/24/2022, 2:29 PM
    thank you
  • n

    noklam

    08/24/2022, 2:40 PM
    I think in Pycharm if u type """ it should just generate the docstring for you?
  • n

    noklam

    08/24/2022, 2:40 PM
    On vscode u have autodocstring
  • d

    datajoely

    08/24/2022, 2:41 PM
    There is a setting to do this in bulk apparently https://stackoverflow.com/questions/37549741/is-it-possible-to-bulk-add-docstrings-to-all-the-functions-in-pycharm
  • b

    Byron

    08/25/2022, 2:40 PM
    Hello guys, I was doing some research about how to run kedro pipelines in containers and I found that the only thing needed besides the dependencies of the project is to have a python virtualenv, am I right or am I missing something?
  • d

    datajoely

    08/25/2022, 3:10 PM
    No that's it! There is also a kedro-docker plugin designed to get you started
  • w

    waylonwalker

    08/27/2022, 8:27 PM
    Nope, that is one of the great benefits of kerdro being written in python. I started with the kedro docker plugin. That gives you a good starting point for your docker journey, but you might need a few specific things for your environment. Such as system dependencies or database drivers. Once you have on that works for your environment you probably don't need the kedro docker plugin anymore unless you prefer the convenience of kedro cli.
  • t

    Thiago Poletto

    08/29/2022, 12:44 PM
    hey guys I'm facing some issues when trying to write data from one BQ table to another, I did set the if_exists: 'append' but when I do it throws error
  • t

    Thiago Poletto

    08/29/2022, 12:45 PM
    message has been deleted
  • d

    datajoely

    08/29/2022, 2:00 PM
    Can you show us your catalog entry? To may have to put it under your save_args
  • s

    sri

    08/29/2022, 7:07 PM
    I have two pipelines for: monthlypipeline = daily_data_pipeline + monthlyaggregate_pipe weeklypipeline = daily_data_pipeline +weeklyaggregate_pipe Since the pipelines share nodes, i get error. what is the best way to write this in kedro?
  • n

    noklam

    08/30/2022, 9:48 AM
    What kind of error are you getting? These kinds of dependencies are what Kedro is designed for, it should resolves the DAGs and merged any repeated nodes automatically.
  • s

    sri

    08/30/2022, 2:40 PM
    i get error for "Pipeline nodes must have unique names". what is the best way to handle this? any sample code
  • d

    datajoely

    08/30/2022, 3:29 PM
    Provide a variation of the
    name
    arg, or actually drop the explicit name altogether
  • t

    Thiago Poletto

    08/31/2022, 11:45 AM
    sorry answering you just now, I did check the save args and now its working, I just miss place a args...
  • j

    Jose Alejandro M

    08/31/2022, 10:17 PM
    Hi, i would like to ask you something guys ... i have a question regarding the execution of the nodes. nodes that come from the same dataset are executed one after the other as if they were in a serial pipeline?, if so, is there a way to open a thread for each process to make it go faster? Here i attach some pictures to have a reference of what i am talking about
  • n

    noklam

    08/31/2022, 10:20 PM
    Look for runner, by default it uses a sequentialrunner, there are parallelrunner and threadrunner as well.
Powered by Linen
Title
n

noklam

08/31/2022, 10:20 PM
Look for runner, by default it uses a sequentialrunner, there are parallelrunner and threadrunner as well.
View count: 1