778216384475693066 #advanced-need-help

Channels

advanced-need-help

job-posting

welcome

noklam

07/08/2022, 4:34 PM

@Siavash Are you using modular pipeline already? https://kedro.readthedocs.io/en/stable/nodes_and_pipelines/modular_pipelines.html

deepyaman

07/08/2022, 6:24 PM

Are you using Kubeflow Pipelines? To build on what @noklam said, I would recommend defining (Python-based) components that run each modular pipeline, and passing artifacts between them. Your containers can either get created by passing

packages_to_install

based on the requirements for the modular pipelines, or you can just package the modular pipeline itself and build based on that. Most of my understanding is of the Kubeflow v2 SDK in Python (which is still in beta), but you can achieve similar results in earlier versions I believe.

deepyaman

07/08/2022, 6:26 PM

Also, ~~we are forced to use Kubeflow.~~ we have the opportunity to use Kubeflow. Think positive. 😛

user

07/10/2022, 4:26 PM

Looking for the right way to make a kedro node output lazily two partitioned datasets https://stackoverflow.com/questions/72930056/looking-for-the-right-way-to-make-a-kedro-node-output-lazily-two-partitioned-dat

venncit

07/10/2022, 8:10 PM

Hi! Has anyone experience running kedro (0.17.5) on Amazon Managed Workflows for Apache Airflow? Once all kedro project (as wheel in plugins.zip) and python dependencies are installed, the first node (KedroOperator) runs infinite and but won't execute. Used kedro-airflow package to create DAGs. It's stuck on

{{standard_task_runner.py:52}} INFO - Started process 227 to run task

and seems like a setting/dependency is not properly met for some reason.

noklam

07/10/2022, 8:12 PM

Can you tried add some priting statement in the KedroOperator to see where did it stuck?

noklam

07/10/2022, 8:12 PM

Does it fails or it just doesn't execute at all?

venncit

07/10/2022, 8:42 PM

When I put a print statement right after

def execute(self, context):

it won't even print it

venncit

07/10/2022, 8:42 PM

It won't fail, it remains in running state

venncit

07/10/2022, 8:45 PM

Copy code

*** Reading remote log from Cloudwatch log_group: airflow-Airflow-FCC-Task log_stream: ads-f9329-forecast-customer-care/feature-engineering-prophet-node/2022-07-10T20_40_29.715933+00_00/1.log.
[2022-07-10 20:40:30,595] {{taskinstance.py:877}} INFO - Dependencies all met for <TaskInstance: ads-f9329-forecast-customer-care.feature-engineering-prophet-node 2022-07-10T20:40:29.715933+00:00 [queued]>
--------------------------------------------------------------------------------
[2022-07-10 20:40:30,622] {{taskinstance.py:1089}} INFO - Executing <Task(KedroOperator): feature-engineering-prophet-node> on 2022-07-10T20:40:29.715933+00:00
[2022-07-10 20:40:30,625] {{standard_task_runner.py:52}} INFO - Started process 314 to run task

venncit

07/11/2022, 9:15 PM

Got it working on MWAA thanks to this: https://github.com/astronomer/cs-tutorial-kedro/blob/main/include/kedro_lib.py. Seems python3.7 requires something special...

Matthias Roels

07/12/2022, 6:18 PM

Not sure if this is the right medium/channel for my question, but I have a rather specific and advanced question on upgrading to kedro 0.18.2. The registration hooks are removed from the code-base, but we had a specific use-case where we need access to the ConfigLoader object when registering pipelines. We did that by adding the ConfigLoader object as an attribute of our registration hooks class. Is it possible to do something similar with the new registration mechanism?

datajoely

07/12/2022, 6:19 PM

What were you trying to do? Provide the

env

attribute is a common use case here

Matthias Roels

07/12/2022, 6:37 PM

If I would be able to get that already would be great. But I don’t see how…

datajoely

07/12/2022, 6:39 PM

This is the way to do that in 0.18.x https://discord.com/channels/778216384475693066/778998585454755870/980886228700385330

Matthias Roels

07/12/2022, 7:02 PM

Yes I know this is how to overwrite the ConfigLoader, but how to make sure you have the

env

attribute available in register_pipelines?

datajoely

07/12/2022, 7:26 PM

There is no register pipelines anymore, since they're python packages now they're importable and the hooks are redundant. You can maybe use a before_pipeline_run hook?

Matthias Roels

07/12/2022, 7:40 PM

And what does this do: https://github.com/kedro-org/kedro/blob/main/kedro/templates/project/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/pipeline_registry.py ?

datajoely

07/13/2022, 9:20 AM

The pipeline registry is the modern way to register pipelines. You can read about this in the upgrade guide in our realease notes.

Matthias Roels

07/13/2022, 9:30 AM

Yes, I know! But I want to have the

config_loader

object (or the

env

var) available in

register_pipelines

and I have no idea how (as the

register_pipelines

does not accept any argument)

datajoely

07/13/2022, 10:04 AM

So it would be useful to know what you're planning to do with this info?

Matthias Roels

07/13/2022, 10:04 AM

The reason I'm asking is that we have a use-case where we defined all our pipelines (and nodes) in yaml, following the same structure as our parameter/catalog config (using env to allow pipeline/node overwrites). In v0.16.x: we could facilitate that through a CustomContext In v0.17.x: we implemented that through a RegistrationHooks class containing register_config_loader (setting

self.config_loader

) and register_pipeline hook (that leverages

self.config_loader

)

Matthias Roels

07/13/2022, 10:05 AM

I think internally, this is known as kedro-glass, no?

datajoely

07/13/2022, 10:05 AM

So the kedro maintainer team aren't really fans of this approach

Matthias Roels

07/13/2022, 10:05 AM

We are aware of that...

datajoely

07/13/2022, 10:05 AM

but I'll look into how this works

Matthias Roels

07/13/2022, 10:06 AM

Thanks a lot!

Matthias Roels

07/13/2022, 10:06 AM

Tbh, this is a setup we inherited from QB consultants and it turns out to be really difficult to change

datajoely

07/13/2022, 10:07 AM

understood

Matthias Roels

07/13/2022, 10:08 AM

As a rather hacky solution, I could already have the

env

set as an actual environment variable (

KEDRO_ENV

) and use that in my custom

register_pipelines

function to create a new instance of

config_loader

and fetch my pipelines from there. But it is not an ideal work around...