It's my own project

basically I've got a few pipeline that are disconnected with output O-1 of pipeline 1 being named differently from input I-2 of pipeline 2. Therefore I created a pipeline with pipe = pipeline1, and inputs={'I-2-name':'O-1-name'}

Hi team, I have a quick question about the partial parallel pipeline. I'm thinking of a pipeline: node_1 -> node_2 -> (parallel) node_3_1, node_3_2 is this possible to define in pipeline.py?

Perfect, thank you! That was the confirmation I was looking for! 😄 I guess the confusion came from the fact that I usually install all dependencies at once but now it's a two-step process. Thanks again!

Hello I've been working a bit with kedro and installed it on python 3.9, I have had no problems so far. When loading in custom datasets there is a warning : DeprecationWarning: The transformer API will be deprecated in Kedro 0.18.0.Please use Dataset Hooks to customise the load and save methods. I want to make my code future proof so I was looking in to creating hooks for my custom datasets. I looked at https://kedro.readthedocs.io/en/stable/07_extend_kedro/02_hooks.html#use-hooks-to-customise-the-dataset-load-and-save-methods but it was not clear to me how this can overwrite the save and load method of multiple custom datasets. How would you implement a hook based on a specific datatype?

Hi @User I think there is a chance we're over-warning in this case. Unless you're implementing a hook that overrides the

load

or

save

dataset methods (like the expanded example here https://github.com/kedro-org/kedro/blob/e78990c6b606a27830f0d502afa0f639c0830950/docs/source/07_extend_kedro/06_transformers.md) I don't think you have to worry

Ah ok, great. Thanks for the quick response! 😀

hello yall, I would like to change path in comment when running 'kedro build-reqs'. In my requirements.txt, I would like than the line begin from my source route (src). Any suggestions so line 5 look like 6 or 7 ?

Hello! Good question and I think there is probably scope for us to make this more configurable

Looking at the code here (https://github.com/kedro-org/kedro/blob/a839d6952a88414234d25d6df09422eaf2260011/kedro/framework/cli/project.py#L73) it looks like we just run the following command (

-q

puts it in less verbose mode), but the location is hard coded

pip-compile -q requirements.in

I think its easiest to not use the

kedro

command wrapper here and use

pip-compile

directly, if you really want to you can define your own

cli.py

and your local version of the command will take precedence.

alright thank you for your answer

Is there a well-defined pattern for bubbling kedro pipelines up through dev/qa/prod environments? My specific use case is in the intermediate steps where data is serialized after any given node. We save parquet files in s3 however as we move pipelines out of dev and into a qa or prod environment we'd have to change those s3 bucket paths. I'd like to be able to configure the bucket based on an environment variable but this isn't immediately obvious to me as to how to do it without hijacking the config loader. Am I missing a simple answer?

Yes we have exactly that

I'll read the friendly manual! Thanks! Tagging @waylonwalker for work tomorrow

So I have a PatitionedDataSet that’s comprised of ‘.xls’ files. I’m trying to parse a subset of the columns as datetime[ns] types since they are coming in as objects. How would I denote that in the catalog? Presently, I have the data set load_args set with the following: load_args: dtype: {‘a’:’datetime[ns]’} I’m not getting an error but the types aren’t conforming to date times for the columns I’ve specified.

so

pd.read_excel

is expecting a python type, but Kedro is providing a

str

we don't have a great/safe way of providing proper python types from YAML here

so you have two options (1) You can convert types in the node (2) You can define a custom dataset to do this in the YAML definition

I think I’ll go with the former for now. Thanks!

You can use parameters here if you want to keep the definition declarative

So if I used the parameters.yml, would it be something like dtype: “params:datetimes”

If you

datetimes

key included a list of columns I would map a dictionary of column name to date type pairs and pass those to

df.astype

Maybe it's me being too simple but I have never used load_args.... even before kedro I never used those arguments. It feels like one more thing to know rather than just using astype, which you are going to need to know and use anyways.

So there is a performance benefit in many cases doing it on load, given were using xls it's not going to be significant

I could see that, almost all of my pipelines run on a schedule, for pennies per run.... maybe that privilege of never worrying about pipeine perf has me spoiled. readability, maintainablility >> perf for a pipeline that runs for a few minutes when no one is looking.

I think that's valid, but I would also say there is a ton of functionality in the

load_args

and

save_args

that you should be aware of and in some cases keep declarative and configurable in the catalog

Hi ! actually i'm strugling with a resources issues, because i have to process a lot of images with high resolution and the node processing have to store them in RAM and that raises OutOfMemory issue. so i think of processing that sequentially in the node ( the node process image then save it, not like now all images are processed then saved in catalog). i found that it can be doable with custom runner isn't it ? any tips for that ?

Releasing memory deliberately

Hello, I tried to install kedro great but I get an error on a lib. It is still maintain ?