👍

when using pandas.CSVDataSet with no load_args, saving and then reading a dataset (both in a pipeline with it once as an 'output' followed by being an 'input', and just in a notebook or ipython session), if the csv contains é, it fails.

One must specify 'my_csv: load_args: encoding: latin1' in the catalog. Specifying an encoding in 'my_csv: save_args: encoding' doesn't seem to have any impact on the encoding that pandas uses. If you specify the same encoding in both save_args and load_args, it will still write to 'latin1'

Just discovered Kedro recently and would like to use it for a project. Problem is, that I have a hard time finding a working configuration example for a certain data access mode I need: Reading partitioned pandas.CSVdatasets over SFTP

Would anybody be able to provide a working example or pointer where to look for the right fsspec-parameter to place in base/catalog.yml and whatnot?

A secondary question would be, how to specify that I only want files considered that match a certain pattern, like /my/path/to/data/this_is_it*.csv

I gather from the documentation that specifying the directory and the .csv suffix is not a problem, but I can't find the spot to place the "this_is_it*" part to filter out the correct files.

@User you're in luck (partially, at least) since an example of

pandas.CSVDataSet

over sftp was just added to our docs a couple of days ago - see Example 16 here: https://kedro.readthedocs.io/en/latest/05_data/01_data_catalog.html#using-the-data-catalog-with-the-yaml-api It should be straightforward to combine this with

PartitionedDataSet

by following the example here https://kedro.readthedocs.io/en/latest/kedro.io.PartitionedDataSet.html#kedro.io.PartitionedDataSet. You'd end up with something like this:

my_dataset:
  credentials: cluster_credentials
  path: sftp:///path/to/remote_cluster/
  filename_suffix: .csv
  type: PartitionedDataSet
  dataset:
    type: pandas.CSVDataSet

As to pattern matching instead of just suffix matching, this is not immediately possible. I had the idea a while ago to change

filename_suffix

to a more general

filename_pattern

but I don't think it ever got implemented. If there's interest in adding this we can certainly think again about it. For now, your options would be: * change your filenames to match a suffix pattern like *_this_is_it.csv (note in can be more than just the .csv file extension) * subclass

PartitionedDataSet

to make your own custom dataset type that implements

filename_pattern

instead of

filename_suffix

. This would be straightforward but a bit annoying given

partition.endswith(self._filename_suffix)

seems to appear in several methods

@User what version of pandas and kedro are you using? I remember spending a while investigating this before since the behaviour has changed... Possibly you will need to specify encoding through

fs_args: open_args_save: encoding

(and similarly for

load

).

Hi! I have two questions: 1- what is the best practice concerning where to place custom hooks in a kedro project or where are you generally saving them? 2- if you have a project with more than one pipeline in it and would like to have hooks that run only with a specific pipeline how would you implement this ?

Hello @User ! 1. I would go for

src/project_name/hooks.py

. If you have a lot of hooks then you might like to make a new package

src/project_name/hooks/

and break them into multiple files in there

(N.B. in kedro 0.18 this file won't exist by default any more, so you'll need to make it yourself. It's still the right place to put project-specific hooks though)

2. This depends a bit on which hooks you're trying to use. Pipeline hooks (i.e.

before_pipeline_run

,

after_pipeline_run

,

on_pipeline_error

) have both

run_params

and

pipeline

available to them as arguments. Hence you could do a conditional hook like this:

class Hooks:
    @hook_impl
    def before_pipeline_run(self, run_params: Dict[str, Any]) -> None:
        if run_params["pipeline_name"] == "data_science":
             # code that only runs when you call `kedro run --pipeline=data_science`

Alternatively you can use

pipeline

which will contain the actual

Pipeline

object (i.e. collection of nodes) that kedro is going to execute in that

kedro run

Now if you want to make a hooks conditional on a particular pipeline but it's not a pipeline hook (e.g.

before_node_run

) then you'll need to make the pipeline name accessible to your hook by saving the relevant information to the class somehow, e.g.

class Hooks:
    @hook_impl
    def before_pipeline_run(self, run_params: Dict[str, Any]) -> None:
        self.pipeline_name = run_params["pipeline_name"]

    @hook_impl
    def before_node_run(self) -> None:
        if self.pipeline_name == "data_science":
            # code that only runs when you call `kedro run --pipeline=data_science`

Thanks a lot for your reply !! The idea of saving the run params to make them available to the before_node_run is a cool idea ! Thanks !

Kedro 0.17.6, pandas 1.3.5, probably too new a version of pandas as I’m stuck with python 3.9 on just the one machine with data. I’ll try again with a fresh install and 3.8 and let you know if it’s still a problem. Thanks

Hello, beginner here. I wanted to use hive context inside my node. I have enabled hive support in building spark session. Then inside the node I tried calling spark session:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

df = spark.sql("SELECT * FROM ....")

But I got the error:

ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1641884194120_477874 to YARN : Application rejected by queue placement policy

Spark

Hi All! I was running tests locally and facing failed cases for just single file: tests/framework/cli/pipeline/test_pipeline_requirements.py Is this just for me?

what version of python are you runnung?

3.8.12

Is it just me? @User

I'm not sure - can you paste the error?

and are you on Windows?

Wait let me check

Is this what you are looking for?

This is going to be something wrong with your local environment 🤔 Please can you do

pip show setuptools pip

and paste it here?

This file passes tests for me locally and on CI so you should be able to get it to pass with the right environment

Okay let me see