https://kedro.org/ logo
Join the conversationJoin Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
beginners-need-help
  • c

    ChainYo

    02/05/2022, 11:47 AM
    I am having trouble with Minio credentials stored in
    credentials.yml
    . It doesn't work and it seems that the credentials aren't used like it should be. The strange thing is I use also credentials for another modular pipeline before the one with Minio but with Binance credentials and it works. I'm confused about the error I got because I do exactly the same process
  • c

    ChainYo

    02/05/2022, 11:47 AM
    message has been deleted
  • c

    ChainYo

    02/05/2022, 11:49 AM
    Is there a problem between params type given by kedro credentials thing and minio expected inputs ?
  • c

    ChainYo

    02/05/2022, 12:22 PM
    Btw when I directly give Minio client the credentials in the python script it works, but I don't really want them written in the file πŸ˜„
  • c

    ChainYo

    02/05/2022, 12:33 PM
    Is it possible that credentials are loaded only for the first modular pipeline and not for all modular pipelines during the run ?
  • b

    beats-like-a-helix

    02/05/2022, 10:19 PM
    Hello there! I'm new to Kedro. I was reading the release notes for 0.18 and noticed
    kedro install
    is being depreciated in favour of
    pip install -r src/requirements.txt
    to install dependencies. Does this effectively mean that managing dependencies via conda alone is now depreciated, given that
    kedro install
    currently supports installing from
    src/environment.yml
    ?
    d
    c
    • 3
    • 10
  • a

    avan-sh

    02/05/2022, 11:02 PM
    @User I'm not 100% sure[just another user], but I think the main motto here is letting the users handle their environment. So you'll likely be free to choose between various dependency management tools, conda/pip or anything else .
  • a

    avan-sh

    02/06/2022, 12:00 AM
    Hi @User , If I understand right, you're reading credentials as parameters directly on the node input and initiating a client to read/write files. I don't think credentials could be read as params, unless I'm missing some extra magic in between. An alternate approach might be to use Example: 14 from https://kedro.readthedocs.io/en/stable/05_data/01_data_catalog.html#using-the-data-catalog-with-the-yaml-api
  • c

    ChainYo

    02/06/2022, 7:51 AM
    I will try the example but it seems to be for defining a dataset in the catalog using minio as source. In my case I don't use minio as source but only as target πŸ™‚
  • c

    ChainYo

    02/06/2022, 7:52 AM
    I did this for an API client on the first node of the first modular pipeline and it worked well πŸ™‚
  • c

    ChainYo

    02/06/2022, 7:53 AM
    That's why I'm surprised it doesn't work for another node on anothe modular pipeline of the same project
  • a

    avan-sh

    02/06/2022, 8:09 AM
    The same should work even for writing to minio. I tested it locally, but I was using access keys instead.
  • c

    ChainYo

    02/06/2022, 8:10 AM
    what do you give to him inside the client ?
  • c

    ChainYo

    02/06/2022, 8:11 AM
    client = Minio(**minio_creds)
  • a

    avan-sh

    02/06/2022, 8:14 AM
    You won't need to create a client at all in the node. Ideally nodes should only contain any transformation logic. The dataset abstraction is available so that you can just change the destination/source filesystems without having to change your pipelines at all.
  • a

    avan-sh

    02/06/2022, 8:15 AM
    companies: type: pandas.CSVDataSet filepath: s3://test-bucket/primary/companies.csv credentials: dev_minio preprocessed_companies: type: pandas.CSVDataSet filepath: s3://test-bucket/primary/preprocessed_companies.csv credentials: dev_minio
  • c

    ChainYo

    02/06/2022, 8:15 AM
    Thx for the hint
  • b

    beats-like-a-helix

    02/06/2022, 5:41 PM
    Interesting use case, I'm also a python-binance user. I take it you're using Kedro purely for analysis rather than executing trades?
  • b

    beats-like-a-helix

    02/06/2022, 5:49 PM
    Another general question: it's clear that the design of Kedro is quite heavily inspired around ML use cases. Is it still advised to use Kedro for data science projects that have no ML involved?
    d
    • 2
    • 3
  • c

    ChainYo

    02/06/2022, 5:52 PM
    I'm using it to automatically train then serve LSTM models
  • c

    ChainYo

    02/06/2022, 5:53 PM
    And I'm using Binance API to feed data for training and inference
  • b

    beats-like-a-helix

    02/06/2022, 5:59 PM
    Very cool! I've been wanting to do something similar, but it'll be summer before I have the time
  • d

    datajoely

    02/06/2022, 7:27 PM
    Deprecation of
    kedro install
  • d

    datajoely

    02/06/2022, 7:30 PM
    Kedro without ML
  • d

    datajoely

    02/07/2022, 3:00 PM
    Thanks for helping out the community @User ! You have been upgraded to status πŸ™‚
  • m

    martinlarsalbert

    02/08/2022, 7:24 AM
    with kedro.extras.datasets.matplotlib.matplotlib_writer I sometimes get the following error: Fail to create pixmap with Tk_GetPixmap in TkImgPhotoInstanceSetSize, anyone recognize this error and how to resolve it?
  • d

    datajoely

    02/08/2022, 9:41 AM
    As far as I can work out - it's a Matplotlib error not a Kedro one as we're just passing things to that API behind the scenes
  • m

    martinlarsalbert

    02/08/2022, 3:48 PM
    Yea definitely a matplotlib error, I just wanted to see if anyone else had seen the same thing with kedro, but clearly not. Thanks anyway.
  • a

    austin-hilberg

    02/08/2022, 9:28 PM
    I've created a new conda environment with
    python=3.8
    , and I've installed kedro and jupyter using
    -c conda-forge
    . However, I'm getting a "no such command 'jupyter'" error. I've set up Kedro and Jupyter many times before in conda environments, and I can't figure out what I'm doing wrong. Thanks!
  • d

    datajoely

    02/08/2022, 9:35 PM
    There’s a chance this conversation is relevant In case useful there is a conda related discussion going on here https://github.com/kedro-org/kedro/discussions/1220
Powered by Linen
Title
d

datajoely

02/08/2022, 9:35 PM
There’s a chance this conversation is relevant In case useful there is a conda related discussion going on here https://github.com/kedro-org/kedro/discussions/1220
View count: 1