https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
advanced-need-help
  • d

    datajoely

    07/15/2021, 12:52 PM
    We're aware of these people https://tokern.io/data-lineage/ and have something internally the does it for Spark
  • m

    Mad Hatter

    07/15/2021, 12:55 PM
    i am working on teradata lineage and i am also including the join types join columns and filter columns once i get approval or rejection i will share the details with you
  • n

    noklam

    07/16/2021, 5:08 AM
    Interesting! I have thought about using query log to build those lineage data and usage patterns
  • n

    noklam

    07/16/2021, 5:08 AM
    did not know these tool exists
  • s

    sigma

    07/16/2021, 2:00 PM
    Though it took about a min to render some 1100 nodes, my firefox was working fine.
  • s

    sigma

    07/16/2021, 2:01 PM
    It might be your system specs that are limiting the performance
  • w

    waylonwalker

    07/23/2021, 12:51 PM
    @User is there any way to silence the kedro_telemetry logger without uninstalling it? Should we be able to for plugins wanting a clean output like
    kedro-diff
    ? Or is more ethical to leave that one on.
  • d

    datajoely

    07/23/2021, 12:54 PM
    Let me check with the team - we should provide a way of doing this
  • w

    waylonwalker

    07/23/2021, 12:56 PM
    sounds good, I think it's getting called before the first line of my cli even fires, so I don't think I have control as a cli plugin author
  • l

    Lorena

    07/23/2021, 3:41 PM
    @User can you set a logging configuration that just mutes the module that logs the telemetry message? Similar to this example: https://github.com/quantumblacklabs/kedro/blob/master/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml#L43-L47
  • w

    waylonwalker

    07/23/2021, 6:08 PM
    @User I cannot do it as the plugin author. It looks like its up to the project author.
  • d

    datajoely

    07/26/2021, 12:31 PM
    @User I was thinking can you retrieve the logger and suppress it this way? https://stackoverflow.com/a/53250066/2010808
  • w

    waylonwalker

    07/26/2021, 12:44 PM
    Kedro diff already silences most of kedro's logs unless you have specified verbose mode. I think the issue is that my silence_logger command happens after the kedro_telemetry.plugin fires, so I cant't silence it before then. I also have to know the name of the logger ahead of time as the logger will be created and fire nearly together.
  • w

    waylonwalker

    07/26/2021, 12:45 PM
    Here is a list of the loggers containing the name kedro in my current session. kedro telemetry did fire, but I do not see it registered as a logger.
  • d

    datajoely

    07/26/2021, 12:45 PM
    Okay good to know - I've added a ticket to our backlog
  • u

    user

    07/28/2021, 5:38 PM
    Sorry I'm late to the party. @User if you open http://127.0.0.1:4141/redoc after running
    kedro viz
    you will see our OpenAPI schema and the Redoc documenation. Please shout if you need help integrating Viz into your lineage export. Indeed we can and should turn viz into a lineage tool, because if you toggle off all of the nodes, you get a DAG of data flow. In fact, Kedro pipelines are controlled by data topology by connecting nodes through their inputs and outputs, so it's a DAG of data, not a DAG of tasks like other workflow engine. In a sense, that's already a data lineage. If we get data schema and column level lineage, we can also render that. But extracting column level lineage in a heterogenous execution environment is difficult. We could do that for Spark, SQL, etc. independently but not when they are all mixed together in a pipeline.
  • d

    datajoely

    07/28/2021, 6:42 PM
    @User FYI
  • s

    sigma

    07/28/2021, 6:45 PM
    Thanks
  • d

    datajoely

    07/28/2021, 6:47 PM
    I should say that this for the latest version of viz released today
  • d

    datajoely

    07/28/2021, 6:47 PM
    https://discord.com/channels/778216384475693066/778996598880862208/869992361361571840
  • w

    waylonwalker

    08/02/2021, 1:43 PM
    Can we get a pypi framework :: kedro??? I tried to add it to a package, I think it needs to exist before I can publish with that classifier.
  • w

    waylonwalker

    08/02/2021, 1:44 PM
    Here is a list of all pypi classifiers. https://pypi.org/classifiers/
  • d

    datajoely

    08/02/2021, 1:45 PM
    Some people are already doing this https://pypi.org/search/?c=Framework+%3A%3A+Kedro
  • w

    waylonwalker

    08/02/2021, 1:46 PM
    huh, wonder why this was not working for me, I'll look at those examples
  • w

    waylonwalker

    08/02/2021, 1:47 PM
    I had lowercase kedro
  • w

    waylonwalker

    08/02/2021, 2:11 PM
    kedro-diff is now on the list. Thanks for debugging with me @User .
  • w

    waylonwalker

    08/11/2021, 2:04 PM
    A potential issue with
    kedro-diff
    is, if you have large files in your project that are not required for making the pipeline objects it can fill up your tmp directory. I think the solution is to implement a .kedroignore, that will ignore certain files and directories specified by the user. If one does not exist, I will ignore the following items by default.
    python
      default_ignore_items = [".envrc", ".venv", ".kedro-diff", "data"]
    What else belongs in the default ignore? are there large files stored in common directories that we dont need while running diff?
  • a

    Arnaldo

    08/11/2021, 2:11 PM
    maybe
    docs
    and
    logs
    folders as well, @User
  • w

    waylonwalker

    08/11/2021, 2:11 PM
    good point
  • u

    user

    08/16/2021, 2:01 AM
    Building an autoencoder with Keras and Kedro https://stackoverflow.com/questions/68796641/building-an-autoencoder-with-keras-and-kedro
Powered by Linen
Title
u

user

08/16/2021, 2:01 AM
Building an autoencoder with Keras and Kedro https://stackoverflow.com/questions/68796641/building-an-autoencoder-with-keras-and-kedro
View count: 1