https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • u

    user

    06/18/2021, 4:40 PM
    This is the best place for me. Waiting for more content 😄
  • d

    datajoely

    06/18/2021, 4:58 PM
    What would you like see more of @User ?
  • u

    user

    06/18/2021, 6:17 PM
    Ideally some working workflow a full process, on the infographic or flowchart to understand the logic and have a big picture of the process and use case at the end.
  • w

    waylonwalker

    06/18/2021, 7:50 PM
    Have you tried the kedro spaceflights tutorial?
  • y

    Yetunde

    06/20/2021, 10:49 PM
    I'm just going to point you to the Kedro Spaceflights tutorial that @User mentioned: https://kedro.readthedocs.io/en/stable/03_tutorial/01_spaceflights_tutorial.html Try this and let us know how you get on.
  • u

    user

    06/21/2021, 6:52 AM
    Ok I will, thank you
  • d

    datajoely

    06/21/2021, 8:16 AM
    This walkthrough may also be helpful 🙂 https://dev.to/waylonwalker/kedro-spaceflights-part-1-2o3p
  • w

    waylonwalker

    06/21/2021, 1:31 PM
    I have both part 1 and 2 on Youtube as well.

    https://www.youtube.com/watch?v=Y07UBr9Ccjsâ–¾

    https://www.youtube.com/watch?v=_7MwgKu-844â–¾

  • d

    DhruvK

    06/21/2021, 5:04 PM
    Does kedro docker lib only for specific for kedro projects? Or can It be utilised to deploy a flask ml api?
  • d

    datajoely

    06/21/2021, 5:05 PM
    It's very much written with Kedro in mind - especially for users who don't know much about docker. So it's not hard to do from scratch, but it does most of the Kedro stuff for you e.g. the Viz component have all the ports set up nicely.
  • d

    DhruvK

    06/21/2021, 5:07 PM
    Ahh , I seee. I tried automating the same stuff few months bck, 5-6 commands to create container but my code use to take requirements.txt and docker file. I am curious how you guys are generating it.
  • d

    datajoely

    06/21/2021, 5:08 PM
    Here is the source code - https://github.com/quantumblacklabs/kedro-docker/blob/develop/kedro_docker/plugin.py - it is mostly just generating docker stuff for you
  • d

    DhruvK

    06/21/2021, 5:18 PM
    Thanks, I see, you guys are not utilising the database or not interacting with any databases or any cloud infrastructure except dolt. I faced lot of challenges to install configurations for database setup inside container. Most of ml/ds use cases interact with these technologies.
  • d

    datajoely

    06/21/2021, 5:19 PM
    Hi @User I'm not sure that's true - give me a sec to point you to the right sections
  • d

    datajoely

    06/21/2021, 5:20 PM
    So we have two ways of interacting with SQL via Pandas/SQLAlchemy: https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.pandas.SQLQueryDataSet.html And via Spark: https://kedro.readthedocs.io/en/stable/kedro.extras.datasets.spark.SparkJDBCDataSet.html
  • d

    datajoely

    06/21/2021, 5:21 PM
    The point is Kedro doesn't re-invent the wheel it wraps technologies like Pandas and Spark to do the actual database communication, but provides a YAML API for declaring different catalog entries
  • d

    datajoely

    06/21/2021, 5:21 PM
    does that make sense?
  • d

    datajoely

    06/21/2021, 5:21 PM
    Each of these connectors, like all of our connectors can be deployed locally or in the cloud
  • d

    DhruvK

    06/21/2021, 5:27 PM
    Yes, it does. This helps in understanding the interaction with different systems of Kedro. I thought you guys are using apache Livy or rest API to trigger jobs on cluster
  • d

    datajoely

    06/21/2021, 5:27 PM
    No - we're much simpler than that
  • d

    datajoely

    06/21/2021, 5:28 PM
    Kedro isn't an orchestrator it's a framework for organising your code when building ML pipelines and models
  • d

    DhruvK

    06/21/2021, 5:28 PM
    Thanks , let me explore more and deep dive in kedro
  • d

    datajoely

    06/21/2021, 5:29 PM
    Keep the questions coming!
  • d

    Deutöic

    06/28/2021, 5:44 PM
    Hi guys, is it possible to run kedropackage using miniconda?
  • d

    datajoely

    06/28/2021, 5:44 PM
    Hey @User ! As in from the CLI use
    kedro package
    ?
  • d

    Deutöic

    06/28/2021, 5:46 PM
    Yeah, I'm having an issue with kedro package, using a python 3.8 from miniconda
  • d

    Deutöic

    06/28/2021, 5:46 PM
    I'll send a screenshot
  • d

    datajoely

    06/28/2021, 5:46 PM
    Sure please do
  • d

    Deutöic

    06/28/2021, 5:47 PM
    2021-06-28 14:41:50,423 - root - INFO - Registered CLI hooks from 1 installed plugin(s): kedro-telemetry-0.1.1 Traceback (most recent call last): File "/home/grover/miniconda3/lib/python3.8/site-packages/kedro/framework/cli/utils.py", line 380, in load_entry_points entry_point_commands.append(entry_point.load()) File "/home/grover/miniconda3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2446, in load self.require(*args, **kwargs) File "/home/grover/miniconda3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2469, in require items = working_set.resolve(reqs, env, installer, extras=self.extras) File "/home/grover/miniconda3/lib/python3.8/site-packages/pkg_resources/__init__.py", line 775, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.VersionConflict: (python-slugify 5.0.2 (/home/grover/miniconda3/lib/python3.8/site-packages), Requirement.parse('python-slugify~=4.0')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/grover/miniconda3/bin/kedro", line 8, in sys.exit(main()) File "/home/grover/miniconda3/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 264, in main cli_collection = KedroCLI(project_path=Path.cwd()) File "/home/grover/miniconda3/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 187, in __init__ ("Project specific commands", self.project_groups), File "/home/grover/miniconda3/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 238, in project_groups plugins = load_entry_points("project") File "/home/grover/miniconda3/lib/python3.8/site-packages/kedro/framework/cli/utils.py", line 382, in load_entry_points raise KedroCliError(f"Loading {name} commands from {entry_point}") from exc kedro.framework.cli.utils.KedroCliError: Loading project commands from airflow = kedro_airflow.plugin:commands
  • d

    Deutöic

    06/28/2021, 5:48 PM
    I'm testing the astronomer airflow deployment
Powered by Linen
Title
d

Deutöic

06/28/2021, 5:48 PM
I'm testing the astronomer airflow deployment
View count: 1