Hi all, I recently discovered Kedro and would love to contribute to this amazing project. As I was going through the recent PRs, I noticed that e2e tests have been failing for all of them. Is this a known issue ?

They’re failing on some windows edge case . We’re working on them, you can ignore for now 🙃

Thanks 👍

No, the "No such file" error was in reference to the kedro "binary" that gets called when you do "kedro run". I tried transferring that file to a known location on the cluster and it executed fine but then failed to "import kedro" because I hadn't installed kedro on the cluster. I then tried creating a new emr cluster with a "pip install kedro" included in a boostrap.sh. With kedro installed on the cluster, I can now "import kedro" but "kedro run" still errors at "import git" inside kedro.framework.cli.starters, so it seems there are still more dependencies to install. What I was wondering is if there is a way to do runs on the remote cluster without having to install all of the dependencies on the cluster? Maybe something like the databricks-connect example but for emr?

kedro on emr

Has anyone experienced Kedro project breaking when adding additional tools to pyproject.toml? I have tried to simply add poetry snippets to it and it seems that kedro no longer recognises project root folder as a kedro project.

What sort of error are you getting?

After adding poetry part to pyproject.toml trying to call

kedro ipython

for example results in

Error: No such command 'ipython'.

and

kedro --help

shows only basic commands like docs new info and starter

interesting - we don't officially support Poetry so I can't speak with much authority. But I can explain a couple of things that may be playing into this

Deleting additional parts from pyproject.toml fixes things so it's replicable

Oh interesting

Kedro absolutely needs these things

that's how we know if we're in a project or not

It's there, at the top of the file

Basically I have automatically generated kedro file with everything correct

Are you comfortable with DMing me your TOML to review?

I think so, I need to obscure project name but altering the string shouldn't have any effects

I won't be able to help much tonight since it's late here in Lodon - but I can pick things up in the morning 🙂

Sure I think it's an interesting case - it doesn't slow me down that much as I can still work on the other parts - just that dependency management isn't implemented

^ solved, syntax issue

It was, sometimes it's the simplest thing

Hi, does anyone know whether there is a way of adding custom_objects while loading

TensorFlowModelDataset

via

catalog.yml

?

Anyone ever used rpy2 with

kedro jupyter

?

Getting

%load_ext rpy2.ipython

2021-10-21 14:39:23,435 - rpy2.rinterface_lib.callbacks - WARNING - R[write to console]: Error in .Primitive("as.environment")("package:utils") : 
  no item called "package:utils" on the search list

Hi, I'm having some issues with saving and loading pandas.CSVDataSet with characters not in ascii but in utf-8. Error message is 'utf-8' codec can't decode: invalid continuation byte. I've tried to add different encodings to save and load args in catalog but nothing seems to work. Reading the original file works fine and getting expected results in kedro jupyter but saving and loading the data breaks in kedro run. Any advice?

Hi @User - I don't think it's possible out of the box via the YAML API. You can see the implementation here https://github.com/quantumblacklabs/kedro/blob/20f836695c2f1e72f262d1747e47b7b7352a4aa0/kedro/extras/datasets/tensorflow/tensorflow_model_dataset.py#L138 . If you want to do this, I think you would have to subclass the out of the box dataset and include this functionality yourself.

Hi @User - this is likely fixed by providing file system arguments like so:

Tried to add the encoding in the fs_args and get the same error message preprocessed_MediaInvestment: type: pandas.CSVDataSet filepath: data/02_intermediate/preprocessed_MediaInvestment.csv save_args: encoding: 'utf-8' index: True load_args: encoding: 'utf-8' fs_args: open_args_load: encoding: 'utf-8' layer: intermediate

How peculiar - to be clear is this file generated as part of a Kedro run or is it from something external?