I am a little new to kedro. Could you please post the code that can do that?

Hello everyone, I'm in the early stage of my Kedro understanding and I am wondering what the best practice is for visualising data? Typically when using notebooks I'll use packages like matplotlib and seaborn but I'm not entirely sure how they fit into the Kedro workflow? Any advice would be appreciated! Thank you, Lawrence

Data viz in kedro

Folks, quick question. Is there a way to use a parameter from the yml files in order to build a pipeline object? For example if

do_step_a=True

in the yaml add the

step_a

node to a pipeline object? My intuition says that this is a counter pattern.

This is indeed a bit of an anti-pattern, although other people have done similarly. The best way to do it would be using an

after_context_created

hook like this: https://github.com/kedro-org/kedro/discussions/1436#discussioncomment-2789761 If it suits your use case, better would be to register two pipelines in `pipeline_registry.py`:

pipeline_1 = pipeline(...)
pipeline_2 = pipeline_1 + [node_a]

and then run the pipeline you want through

kedro run -p

Another alternative is to keep the node in the pipeline but just change the logic inside the node function itself to skip using something like on

if not do_step_a: return

Hello guys, i'm already facing a problem on using Kedro through Pycharm and SSH connexion to cloudera plateform. I create a project with kedro 0.15.9 on Cloudera Plateform and using spark 2.4 and it works fine on the cloudera plateform (Cloudera Data Science Workbench). Now i'm trying to use the project but with using Pycharm and connecting from ssh tunnel to the CDSW. Now i'm facing a problem when a i run the kedro project and this is the error that i get: 2022-07-04 20:02:03,165 - py4j.java_gateway - ERROR - An error occurred while trying to connect to the Java server (127.0.0.1:35327) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 958, in _get_connection connection = self.deque.pop() IndexError: pop from an empty deque During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1096, in start self.socket.connect((self.address, self.port)) ConnectionRefusedError: [Errno 111] Connection refused

So this is a spark configuration issue, you'll have to play around with spark.yml or environment vars to get it working. Additionally we don't see many 0.15.x projects anymore! So we would encourage you to upgrade

I thought also it a configuration problem. I have a restriction for the spark version so the only version of kedro that is working with spark 2.4 is the 0.15

Ah gotcha

It was a memory size of executor problem by increasing it the code work perfectly

Hello guys, I have a question it's possible to save the output of a pipeline at same time in pandas and spark data frame ? Considering that the input of the pipeline is a spark frame.

Read about transcoding here! https://kedro.readthedocs.io/en/stable/data/data_catalog.html#transcoding-datasets

Thank you ! forgot to precise it's version 0.15.9 😅 but i saw this was possible on latest version

Here is the 0.15.9 version of the docs https://kedro.readthedocs.io/en/0.15.9/04_user_guide/04_data_catalog.html#transcoding-datasets

Hello, I would like to know how I may access the catalog of my project from within a Python script. Providing more context: In a Jupyter notebook, if I use the appropriate kedro magic, I can type catalog.load('foo') to load the dataset named 'foo' which is registered in my project's catalog. I would like to do this in a standalone Python script (a .py file).

If you need to do this inside a kedro pipeline, check out the hooks for after_catalog_created or after_context_created

Thanks. I was hoping to do it outside of a kedro pipeline

s

Error when save using pandas.CSVDataset

Greetings, I have the line app: abcdefgh In the conf/local/credentials.yml file of my kedro project. How do I pass the string 'abcdefgh' as an argument to a node from a modular pipeline? I tried setting the 'inputs' kwarg to ['credentials:app'] but kedro tries to find this in the data catalog.

I'm following along the experiment tracking example but after running the pipeline, kedro viz says "You don't have any experiments" and there is no

session_store.db

There are, however, metrics-related files under

09_tracking

. Did I possibly misconfigure something? Also,

$ cat src/settings.py
from kedro_viz.integrations.kedro.sqlite_store import SQLiteStore
from pathlib import Path

SESSION_STORE_CLASS = SQLiteStore
SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}

Credentials should be declared as an attribute of dataset in catalog.yml, is there a a specific reason why you want to pass credentials directly as an argument?

In general your node shouldn't be aware of the I/O

Did you find your sessionstore.db? It should be a file stored locally

My node makes an API call, and the API is its own processing step. The credential I wish to access within the node is the API key. Thus, my credential is not about accessing data per se. Do you still recommend I add the credential to one of the input or output datasets?

No db files are present. I look specifically for

./data/session_store.db

and did a general find for any *.db files. Nothing was found. In case version info helps:

kedro                         0.18.1
kedro-telemetry               0.2.1
kedro-viz                     4.7.1

Does the APIDataSet fit your usecase? Or you could have a custom api dataset that wrap around your logic

Thanks! I didn't know about this dataset type. It doesn't fit my use case directly because I actually make a series of API calls within my node, and running the node alone takes a lot of time. I'll think of a way to wrap around my logic as you suggested.

I have two pipelines named data_processing and edge_processing. I have one output that I want to use in edge_processing but the pipeline automatically looks for the output as edge_processing.file. Even though I specified the file as an output of data_processing. Is there anyway around this?

What do you mean by specified the file as an output of data_processing? What's the command that you used?