https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • w

    waylonwalker

    06/18/2021, 4:38 PM
    What makes a kedro project a kedro project from the perspective of the cli? Does it change on different versions? I am currently in a project and not seeing my project commands.
  • a

    antony.milne

    06/18/2021, 4:51 PM
    These days it's done by existence of pyproject.toml with a table called
    tool.kedro
    - see
    kedro.framework.startup._is_project
  • a

    antony.milne

    06/18/2021, 4:53 PM
    It's been that way since 0.17.0, though 0.17.1 made the check stricter (needed the
    tool.kedro
    section and not just pyproject.toml file)
  • a

    antony.milne

    06/18/2021, 4:54 PM
    Before that I believe it was the existence of .kedro.yml
  • a

    antony.milne

    06/18/2021, 4:55 PM
    But don't hold me on that last part ๐Ÿ˜…
  • w

    waylonwalker

    06/18/2021, 5:14 PM
    Thanks @User , That helps greately. The project I was having issues with dates back to a 14 template and has been upgraded several times. Its missing a pyproject.toml, and does not use a src directory.
  • u

    user

    06/18/2021, 5:19 PM
    hi - I'm trying to run through the "How to integrate Amazon SageMaker into your Kedro pipeline" tutorial here https://kedro.readthedocs.io/en/latest/10_deployment/09_aws_sagemaker.html and running into an issue with version compatibility between s3fs, kedro, and fsspec
  • u

    user

    06/18/2021, 5:21 PM
    kedro >= 0.16.5 gives the error "cannot import name 'setup_logging' from 'fsspec.utils'" when importing s3fs
  • d

    datajoely

    06/18/2021, 5:22 PM
    Hi @User what version of python are you running?
  • d

    datajoely

    06/18/2021, 5:23 PM
    Unfortunately itโ€™s quite late in the day here in London so you may not get a detailed response until Monday. Iโ€™ll be sure to follow up then if you donโ€™t get any other responses tonight.
  • u

    user

    06/18/2021, 5:24 PM
    3.7.10
  • u

    user

    06/18/2021, 5:25 PM
    appreciate it @User . I can raise the issue on GitHub too
  • d

    datajoely

    06/18/2021, 5:25 PM
    This is now the right forum for support! Just a little late in the day ๐Ÿ˜…
  • u

    user

    06/18/2021, 5:28 PM
    no worries, totally understand. thank you!
  • d

    datajoely

    06/21/2021, 9:24 AM
    Hi @User I've replied on GitHub but - thanks for the detailed issue. We think it's related to the the fact
    fsspec
    is using the Calendar versioning system.
  • u

    user

    06/21/2021, 1:16 PM
    thanks so much for the quick reply ๐Ÿ‘
  • a

    Anish Shah @ WANDB

    06/21/2021, 8:49 PM
    Not sure if here is the right place to ask but if I wanted to use the catalog to pull data from ElasticSearch would I be able to use the APIDataSet to pull the dataset or would I have to implement my own Dataset? And if I can pull it using the APIDataset how do would I go about passing in query parameters? Thank you!
  • a

    Arnaldo

    06/21/2021, 8:56 PM
    I might be wrong, but I believe you can use the
    data
    ,
    params
    or
    json
    of the APIDataSet
  • a

    Arnaldo

    06/21/2021, 8:56 PM
    https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.api.APIDataSet.html
  • d

    datajoely

    06/22/2021, 7:58 AM
    Hi @User - Absolutely the right place to ask. You and @User are correct here. - The
    APIDataSet
    is a simple wrapper for the
    requests
    library so that you can use the ES REST API that way. - It may prudent to implement your own dataset which wraps Elastic Search python client, but I'd personally only resort to this if the API mechanism isn't working for you. If you do end up implementing a custom dataset, please do PR it back!
  • u

    user

    06/22/2021, 1:42 PM
    Hi! I initialized a Kedro project with
    kedro new --starter=pandas-iris
    and then I started modifying the
    catalog.yml
    file. In meantime, a new Kedro version came out, which I promptly installed. Is it possible to update my project to the new version, without losing my modifications? Or should I stash the files I changed, initialize the project again with the new Kedro and then apply my modifications?
  • d

    datajoely

    06/22/2021, 1:44 PM
    Honestly, I would start fresh just for simplicity's sake
  • w

    waylonwalker

    06/22/2021, 2:03 PM
    The kedro team has done a really great job of keeping the catalog, node, and pipeline, api consistent. A simple way to upgrade is to render a new template and copy in the files that you have changed.
  • u

    user

    06/22/2021, 3:43 PM
    Will do that, thanks!
  • s

    sigma

    06/24/2021, 8:07 AM
    Hello! I've been using Kedro for a simple data visualization project. It basically involves visualizing a csv dataset (of a graph database containing many source nodes and target nodes ) through Kedro viz.
  • s

    sigma

    06/24/2021, 8:08 AM
    But I'm running into some errors. In my dataset one valid use case is different nodes can point to same targets (outputs) but that is giving me the
    Node outputs must be unique
    error. Can anybody help here?
  • d

    datajoely

    06/24/2021, 8:13 AM
    Hi @sigma since Kedro is a tool for building DAG execution paths we have a hard constraints that the graph must be acyclic
  • d

    datajoely

    06/24/2021, 8:13 AM
    Am I right in saying that you are just using the Viz library without the core side of kedro?
  • s

    sigma

    06/24/2021, 8:19 AM
    yes @User . That's right. I'm just using Kedro-viz by building a pipeline with a dummy function, since all I need is the graph db to be visualized
  • s

    sigma

    06/24/2021, 8:20 AM
    Oh, I see.
Powered by Linen
Title
s

sigma

06/24/2021, 8:20 AM
Oh, I see.
View count: 3