I am having trouble with Minio credentials stored in

credentials.yml

. It doesn't work and it seems that the credentials aren't used like it should be. The strange thing is I use also credentials for another modular pipeline before the one with Minio but with Binance credentials and it works. I'm confused about the error I got because I do exactly the same process

message has been deleted

Is there a problem between params type given by kedro credentials thing and minio expected inputs ?

Btw when I directly give Minio client the credentials in the python script it works, but I don't really want them written in the file 😄

Is it possible that credentials are loaded only for the first modular pipeline and not for all modular pipelines during the run ?

Hello there! I'm new to Kedro. I was reading the release notes for 0.18 and noticed

kedro install

is being depreciated in favour of

pip install -r src/requirements.txt

to install dependencies. Does this effectively mean that managing dependencies via conda alone is now depreciated, given that

kedro install

currently supports installing from

src/environment.yml

?

@User I'm not 100% sure[just another user], but I think the main motto here is letting the users handle their environment. So you'll likely be free to choose between various dependency management tools, conda/pip or anything else .

Hi @User , If I understand right, you're reading credentials as parameters directly on the node input and initiating a client to read/write files. I don't think credentials could be read as params, unless I'm missing some extra magic in between. An alternate approach might be to use Example: 14 from https://kedro.readthedocs.io/en/stable/05_data/01_data_catalog.html#using-the-data-catalog-with-the-yaml-api

I will try the example but it seems to be for defining a dataset in the catalog using minio as source. In my case I don't use minio as source but only as target 🙂

I did this for an API client on the first node of the first modular pipeline and it worked well 🙂

That's why I'm surprised it doesn't work for another node on anothe modular pipeline of the same project

The same should work even for writing to minio. I tested it locally, but I was using access keys instead.

what do you give to him inside the client ?

You won't need to create a client at all in the node. Ideally nodes should only contain any transformation logic. The dataset abstraction is available so that you can just change the destination/source filesystems without having to change your pipelines at all.

companies: type: pandas.CSVDataSet filepath: s3://test-bucket/primary/companies.csv credentials: dev_minio preprocessed_companies: type: pandas.CSVDataSet filepath: s3://test-bucket/primary/preprocessed_companies.csv credentials: dev_minio

Thx for the hint

Interesting use case, I'm also a python-binance user. I take it you're using Kedro purely for analysis rather than executing trades?

Another general question: it's clear that the design of Kedro is quite heavily inspired around ML use cases. Is it still advised to use Kedro for data science projects that have no ML involved?

I'm using it to automatically train then serve LSTM models

And I'm using Binance API to feed data for training and inference

Very cool! I've been wanting to do something similar, but it'll be summer before I have the time

Deprecation of

kedro install

Kedro without ML

Thanks for helping out the community @User ! You have been upgraded to status 🙂

with kedro.extras.datasets.matplotlib.matplotlib_writer I sometimes get the following error: Fail to create pixmap with Tk_GetPixmap in TkImgPhotoInstanceSetSize, anyone recognize this error and how to resolve it?

As far as I can work out - it's a Matplotlib error not a Kedro one as we're just passing things to that API behind the scenes

Yea definitely a matplotlib error, I just wanted to see if anyone else had seen the same thing with kedro, but clearly not. Thanks anyway.

I've created a new conda environment with

python=3.8

, and I've installed kedro and jupyter using

-c conda-forge

. However, I'm getting a "no such command 'jupyter'" error. I've set up Kedro and Jupyter many times before in conda environments, and I can't figure out what I'm doing wrong. Thanks!