Kedro run with versioned dataset on S3 storage

Are there any examples on wrapping a parquet dataset within a PartitionedDataSet?

This has a csv example but its the same thing

I get the following error now: AttributeError: 'Series' object has no attribute 'to_parquet'

That's because you're returning a series not a pandas dataframe from your node

That is strange because printing out the resultant object tells me that it is a Data frame. 2022-06-27 14:11:10,487 - test_pandas_etl.pipelines.etl.nodes - INFO - Length of target is 7551152

If you post the node function syntax we can try and work it out, but double check that the object being returned is definitely a df

Kedro memory error

Hello, I get a segmentation fault when running faiss from within a Kedro node, however this does not happen when run as a standalone script

2022-06-28 13:56:27,146 - kedro.pipeline.node - INFO - Running node: evaluate_mapping_in_embeddings_faiss: run_faiss([es_df,tu_df]) -> [res_faiss]
zsh: segmentation fault  kedro run --tag run_faiss

Segmentation fault has to be coming from a library that uses C underneath. All I can suggest is that you use a debugger and find out where it fails?

Would add sometimes the IDE may contribute to that too, do u have it when running in a terminal?

Turns out it was a conda environment issue after all

Thanks to @datajoely for all of his help!

Hi, how can I call the given environment name in cli, within my script ( this script is a customized data catalog which later be used in project hooks) With many thanks in advance

Could you explain a little more, do you mean

kedro run --env=prod

?

Thanks for your prompt response. I want to use 'prod' name within this customized data catalogs script before it gets loaded

as in you want the env name in scope for the

TemplatedConfigLoader

?

I am a little new to kedro. Could you please post the code that can do that?

Hello everyone, I'm in the early stage of my Kedro understanding and I am wondering what the best practice is for visualising data? Typically when using notebooks I'll use packages like matplotlib and seaborn but I'm not entirely sure how they fit into the Kedro workflow? Any advice would be appreciated! Thank you, Lawrence

Data viz in kedro

Folks, quick question. Is there a way to use a parameter from the yml files in order to build a pipeline object? For example if

do_step_a=True

in the yaml add the

step_a

node to a pipeline object? My intuition says that this is a counter pattern.

This is indeed a bit of an anti-pattern, although other people have done similarly. The best way to do it would be using an

after_context_created

hook like this: https://github.com/kedro-org/kedro/discussions/1436#discussioncomment-2789761 If it suits your use case, better would be to register two pipelines in `pipeline_registry.py`:

pipeline_1 = pipeline(...)
pipeline_2 = pipeline_1 + [node_a]

and then run the pipeline you want through

kedro run -p

Another alternative is to keep the node in the pipeline but just change the logic inside the node function itself to skip using something like on

if not do_step_a: return

Hello guys, i'm already facing a problem on using Kedro through Pycharm and SSH connexion to cloudera plateform. I create a project with kedro 0.15.9 on Cloudera Plateform and using spark 2.4 and it works fine on the cloudera plateform (Cloudera Data Science Workbench). Now i'm trying to use the project but with using Pycharm and connecting from ssh tunnel to the CDSW. Now i'm facing a problem when a i run the kedro project and this is the error that i get: 2022-07-04 20:02:03,165 - py4j.java_gateway - ERROR - An error occurred while trying to connect to the Java server (127.0.0.1:35327) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 958, in _get_connection connection = self.deque.pop() IndexError: pop from an empty deque During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1096, in start self.socket.connect((self.address, self.port)) ConnectionRefusedError: [Errno 111] Connection refused

So this is a spark configuration issue, you'll have to play around with spark.yml or environment vars to get it working. Additionally we don't see many 0.15.x projects anymore! So we would encourage you to upgrade

I thought also it a configuration problem. I have a restriction for the spark version so the only version of kedro that is working with spark 2.4 is the 0.15

Ah gotcha

It was a memory size of executor problem by increasing it the code work perfectly

Hello guys, I have a question it's possible to save the output of a pipeline at same time in pandas and spark data frame ? Considering that the input of the pipeline is a spark frame.