We're aware of these people https://tokern.io/data-lineage/ and have something internally the does it for Spark

i am working on teradata lineage and i am also including the join types join columns and filter columns once i get approval or rejection i will share the details with you

Interesting! I have thought about using query log to build those lineage data and usage patterns

did not know these tool exists

Though it took about a min to render some 1100 nodes, my firefox was working fine.

It might be your system specs that are limiting the performance

@User is there any way to silence the kedro_telemetry logger without uninstalling it? Should we be able to for plugins wanting a clean output like

kedro-diff

? Or is more ethical to leave that one on.

Let me check with the team - we should provide a way of doing this

sounds good, I think it's getting called before the first line of my cli even fires, so I don't think I have control as a cli plugin author

@User can you set a logging configuration that just mutes the module that logs the telemetry message? Similar to this example: https://github.com/quantumblacklabs/kedro/blob/master/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/conf/base/logging.yml#L43-L47

@User I cannot do it as the plugin author. It looks like its up to the project author.

@User I was thinking can you retrieve the logger and suppress it this way? https://stackoverflow.com/a/53250066/2010808

Kedro diff already silences most of kedro's logs unless you have specified verbose mode. I think the issue is that my silence_logger command happens after the kedro_telemetry.plugin fires, so I cant't silence it before then. I also have to know the name of the logger ahead of time as the logger will be created and fire nearly together.

Here is a list of the loggers containing the name kedro in my current session. kedro telemetry did fire, but I do not see it registered as a logger.

Okay good to know - I've added a ticket to our backlog

Sorry I'm late to the party. @User if you open http://127.0.0.1:4141/redoc after running

kedro viz

you will see our OpenAPI schema and the Redoc documenation. Please shout if you need help integrating Viz into your lineage export. Indeed we can and should turn viz into a lineage tool, because if you toggle off all of the nodes, you get a DAG of data flow. In fact, Kedro pipelines are controlled by data topology by connecting nodes through their inputs and outputs, so it's a DAG of data, not a DAG of tasks like other workflow engine. In a sense, that's already a data lineage. If we get data schema and column level lineage, we can also render that. But extracting column level lineage in a heterogenous execution environment is difficult. We could do that for Spark, SQL, etc. independently but not when they are all mixed together in a pipeline.

@User FYI

Thanks

I should say that this for the latest version of viz released today

Can we get a pypi framework :: kedro??? I tried to add it to a package, I think it needs to exist before I can publish with that classifier.

Here is a list of all pypi classifiers. https://pypi.org/classifiers/

Some people are already doing this https://pypi.org/search/?c=Framework+%3A%3A+Kedro

huh, wonder why this was not working for me, I'll look at those examples

I had lowercase kedro

kedro-diff is now on the list. Thanks for debugging with me @User .

A potential issue with

kedro-diff

is, if you have large files in your project that are not required for making the pipeline objects it can fill up your tmp directory. I think the solution is to implement a .kedroignore, that will ignore certain files and directories specified by the user. If one does not exist, I will ignore the following items by default.

python
  default_ignore_items = [".envrc", ".venv", ".kedro-diff", "data"]

What else belongs in the default ignore? are there large files stored in common directories that we dont need while running diff?

maybe

docs

and

logs

folders as well, @User

good point