Can u upgrade kedro-viz?

have done that too, have viz version 5.0.1

Are you using a virtual environment here?

Is there a way to run a DAG so that one or more nodes failing in the middle don't stop the whole pipeline? I'd like it to continue running the nodes that do not depend on the failed nodes, possibly tracking the failed and skipped nodes and throwing an error or log message at the end summarizing what was failed or skipped. I've thought of writing a custom runner for this, but it occurred to me something others may have worked on or been interested in.

On_node_error hook may help here, but may need something more to figure out the dag dependencies

code running in a container

And does a normal kedro run work ok?

And are you actually using HDFS or spark?

kedro pipelines are running without issue, running on spark

Quick question: looking at the source code of kedro, I noticed this line: https://github.com/kedro-org/kedro/blob/0.17.7/kedro/framework/session/session.py#L179. Does that mean that in orde to use kedro, you always need it to be part of a git repo?

@skuma458 no if you use the exp tracking functionality we track the git sha but don't fail without it

Well, it does fail when I use notebooks (I don't use the exp tracking functionality)

Btw: I think you replied to the wrong person :p

Yes I did!

That being said we may have removed the dependency on git later than 0.17.7

Hey folks. How would I run a piece of code whenever I run kedro? I'm thinking of running alembic migrations that way. Is there a proper hook for that?

So am I right in saying the process has nothing to do with your kedro catalog etc and is more about the same time?

yep. More like defining schemas and adding columns to historical tables. I just want the process to be seamless and live in the same repo 😅

Gotcha

So I'd say there is some merit in orchestrating this with something else like Airflow or Argo just because it will be easier to debug isolated issues.... But after_pipeline_run feels like the safest place to do this

got it. Makes sense! I'm just mimicking the pattern for running migrations whenever you start a server in web dev. For me, it makes sense whenever you had long-lasting pipelines that run in a production setting.

Hi Kedro team, I'm really enjoying structuring my projects with Kedro, thank you very much for this great piece of software! I'm curious about the search functionality in kedro viz experiment section. I would like to be able to search in my experiments based on a value of a variable that I am tracking, but it seems that the search is only going thru the experiment metadata. Is there a specific syntax to search thru the tracked data? If not is there a way to programatically add "notes" an experiment? Thanks for the feedback!

So not yet but this is super valuable feedback

Could you raise an issue on the Kedro Viz repo?

Nope: it is still there in 0.18.x https://github.com/kedro-org/kedro/blob/main/kedro/framework/session/session.py#L157. This means that whenever I am not in a repository, I can't even start a kedro session...

What error are you getting we should be catching it?

Let's say I have a modular execution pipeline which is supposed to use a trained model to analyse N datasets. I understand how to create N instances of my modular pipeline (one for each dataset), but I was hoping there was some way to auto-generate catalog entries for these datasets at runtime. ie. I would have

dataset_1_result, dataset_2_result, ..., dataset_N_result

in the catalog without having to write them in the catalog manually. I understand it's just yaml input at the end of the day, and I could just write a hacky script to populate the catalog with these entires, but just wondered if there's a more "Kedronic" way to accomplish this. Cheers!

Before pipeline run hook!

Awesome, not actually got up to speed with hooks yet. Much obliged!