Title
d
Dhaval
12/22/2021, 10:03 AMSo, I'm using this file to use the same pipeline for train and test inputs. There's an error that pops up
Error: Failed to map datasets and/or parameters: traind
datajoely
12/22/2021, 11:33 AMHi @User our modular pipeline docs are currently being overhauled so this should be better explained in the future (You can look at the two open PRs if you want to)
The wrapper method (
from kedro.pipeline.modular_pipeline import pipelineinputsoutputsoutputstraintestdatainputs = {"data": "train|test" }d
Dhaval
12/22/2021, 11:33 AMHi @User , I just did that
And it works
from kedro.pipeline import Pipeline, node
from kedro.pipeline.modular_pipeline import pipeline
from .nodes import (
clean_column_names,
clean_time_cols,
clean_tertiary
)
def create_pipeline(**kwargs):
cleaning_pipe = Pipeline(
[
node(
func=clean_column_names,
inputs="data",
outputs="data_cols_cleaned",
name="data_col_clean",
),
node(
func=clean_time_cols,
inputs="data_cols_cleaned",
outputs="data_time_cleaned",
name="data_time_clean",
),
node(
func=clean_tertiary,
inputs="data_time_cleaned",
outputs="data_cleaned",
name="data_tertiary_clean",
),
]
)
mod_pipe = Pipeline(
[
pipeline(
pipe = cleaning_pipe,
inputs = {"data": "train"},
outputs = {"data_cleaned": "train_cleaned"},
namespace = "train_pipe",
),
pipeline(
pipe = cleaning_pipe,
inputs = {"data": "test"},
outputs = {"data_cleaned": "test_cleaned"},
namespace = "test_pipe",
)
]
)
return mod_piped
datajoely
12/22/2021, 11:33 AMalso using using
kedro viz --autoreloadit really brings it to life
🦾
d
Dhaval
12/22/2021, 11:33 AMI didn't know about this
d
datajoely
12/22/2021, 11:34 AMIt's new! hence us updating the docs
d
Dhaval
12/22/2021, 11:34 AMDoes it automatically see the changes and upload it on the webpage?
d
datajoely
12/22/2021, 11:34 AMyeah so it scans for changes to the codebase and refreshes when it sees it
d
Dhaval
12/22/2021, 11:34 AMThis is great! I'm really excited 🙂
d
datajoely
12/22/2021, 11:34 AMGood luck!
d
Dhaval
12/22/2021, 11:35 AMThanks @User 😁
I'm gonna be a regular here and I want to make use of this tool to it's maximum, so yeah see you around 😆
d
datajoely
12/22/2021, 11:37 AMModular pipelines are the feature I'm most excited about, but we've done a terrible job documenting and marketing them so far. The new Viz update really brings them to life so any feedback on what we could have done better is very much welcome.
d
Dhaval
12/23/2021, 7:53 AM@User Can namespaces be run individually? I'm asking this because in your example for modular-spaceflights you have used namespaces to shrink it down on one single block. I want to run individual pipelines inside these namespaces and don't know how to
d
datajoely
12/23/2021, 12:00 PMNo namespaces can't be run individually, out of the box. Two ways I would recommend -
add tags to the nodes based on the namespace or register the modular pipeline
pipeline_registry.pykedro run --pipeline {name}d
Dhaval
12/23/2021, 3:26 PMWhat would be a wise project decision?
Having multiple modular pipelines setup individually and then merge them all together in the pipeline_registry.py with namespaces or just tag individual nodes with tags whilst keeping the number of modular pipelines to a lesser count?
d
datajoely
12/23/2021, 4:00 PMI think this is very much something for you to decide
the namespaces are independent of how the pipelines get executed
they are just to organise your work
if you look at my demo project...
kedro run --pipeline "Modelling stage"train_evaluation
|_ random_forest
|_ linear_regressionso in this example the top level namespace is equivalent to the registered "Modelling stage" pipeline but serve different purposes
We could also look to add namespaces to the run arguments in the future, this is the first time someone has asked for it
In general we're also looking to overhaul the syntax for how the run command works, so this is useful
d
Dhaval
12/23/2021, 5:20 PMThis is because I'm thinking of creating a project with kind of independent reconfigurable elements and this would definitely help. As of now, the tag argument helps to tackle the problem. Also, glad I could help 😁
@User If my modular pipeline has 2 inputs then how can I map it to the inputs?
I am using
inputs = {"data": "train", "table_name": "train"}kedro.framework.cli.utils.KedroCliError: 'list' object has no attribute 'data_sets'This is the code
d
datajoely
12/29/2021, 11:18 AMHi @User there are a couple of things wrong with your pipeline code, but I'm also not sure where that error is being called from
so first and most important thing - you don't need to retrieve your paramters like tht
to use the
tablesparams:tablesinputregarding the CLI error - could you post more of the stack trace as I'm not sure what is causing that error
d
Dhaval
12/29/2021, 11:32 AMSo to give you a background,
I am trying to generate a profile of the data using Pandas Profiling in a Pipeline
For this, I have 2 datasets:
1. Train
2. Test
For future purposes it can have n number of datasets. But for now we're sticking to these two
The list of tables available can be found in this file which is situated in
conf/base/parameters/raw_data_load.ymlThis file got generated when I created raw_data_load modular pipeline
The aim of this pipeline was to iterate through the tables available in
conf/base/parameters/raw_data_load.ymlnode.pypipeline.pyd
datajoely
12/29/2021, 11:34 AMOkay I would achieve this another way
do you want to profile every dataset or is it just specific ones?
d
Dhaval
12/29/2021, 11:37 AMAll the ones present in the parameters file. For now what is happening is, since train and test are declared in the catalog.yml file
create_descriptives([train,train]) -> json_trainprofile.to_file(f"data/descriptives/{table_name}.html")d
datajoely
12/29/2021, 11:38 AMOkay - we don't encourage you using parameters outside of the default pattern i.e. you shouldn't declare your own config loader outside of the one that exists behind the scenes
so I would do this two different ways
1. You could create a one node
pandas_profilingpython
complete_profiling_pipeline = sum([pipeline(profiler, inputs=ds, outputs=ds+"_profiled") for ds in ["ds1", "ds2", "ds3"]])2. You use Kedro's lifecycle hooks to do your profiling
d
Dhaval
12/29/2021, 11:44 AMOkay, I'll look into this
d
datajoely
12/29/2021, 11:45 AMyeah it's a subtle difference, but we feel dynamic pipelines get a bit messy
d
Dhaval
12/29/2021, 11:45 AMMy aim with this particular project was to have the yml files editable by the non tech folks and not the source code. Do you think that there's some other way?
d
datajoely
12/29/2021, 11:46 AMAs in they decide which datasets get profiled?
and do you want them to run the profiling on demand or as part of a regular run?
d
Dhaval
12/29/2021, 11:47 AMI want to run this particular modular pipeline once they update the raw_data_load.yml file
You mentioned that I should not use the ConfigLoader in my code but if I don't use that then how can I iterate through the entries available in the
raw_data_load.ymld
datajoely
12/29/2021, 11:48 AMOkay maybe I'd suggest a slight tweak then
d
Dhaval
12/29/2021, 11:48 AMtable_pipelines = [
pipeline(
pipe = base_pipeline(),
inputs = {
"data": f"{table_name}",
"table_name": str(f"{table_name}"),
},
outputs = {
"json_data": f"json_{table_name}"
},
namespace = f"{table_name}",
)
for table_name in sql_data
]d
datajoely
12/29/2021, 11:49 AMOkay let's try and get your original approach working
could you post the full stack trace?
d
Dhaval
12/29/2021, 11:49 AMShould I share a Github repo directly with you?
That would be much better to replicate the stack trace
d
datajoely
12/29/2021, 11:50 AMJust paste the stack trace that produced the
kedro.framework.cli.utils.KedroCliError: 'list' object has no attribute 'data_sets'because I'm not sure why that's being called
d
Dhaval
12/29/2021, 11:50 AMThis is done
d
datajoely
12/29/2021, 11:50 AMIs there no more?
d
Dhaval
12/29/2021, 11:51 AMI'll share the repo in 10 mins. Let me just push that, you'll understand it in a much better way
d
datajoely
12/29/2021, 11:51 AMok
d
Dhaval
12/29/2021, 12:04 PMd
datajoely
12/29/2021, 12:05 PMso first suggestion you don't have to do read/write with pandas profiling
you could simply return a Python Dictionary from this function
and save via the kedro
json.JSONDataSetd
Dhaval
12/29/2021, 12:06 PMI am planning to do that. But the html file is important to be saved
Since the code doesn't execute beyond that point
d
datajoely
12/29/2021, 12:07 PMyou can apparently do the
profile.jsonprofile.htmland return both python objects
and write the HTML as
text.TextDataSetthe other question is when you run the pipeline
could you post the full errro
d
Dhaval
12/29/2021, 12:09 PMHere you go
d
datajoely
12/29/2021, 12:09 PM> No such file or directory: 'config_minimal.yml
> No such file or directory: 'data/descriptives/
I think your filepaths are at the wrong working directory
you can breakpoint and work out where you are
d
Dhaval
12/29/2021, 12:10 PMIf you see the create_descriptives function it expects one dataframe and the other one as a string variable, as of now kedro is passing both of these as a catalog entry
d
datajoely
12/29/2021, 12:11 PMah I see what you want
so there is a hacky way of donig this - but you're going quite far off piste
d
Dhaval
12/29/2021, 12:11 PMIf you check the io.catalog too there train is loaded 2 times as well
I just want to pass the string value
d
datajoely
12/29/2021, 12:12 PMd
Dhaval
12/29/2021, 12:12 PMAlso how do i replace the sql_data with params, because that's where my input tables are
d
datajoely
12/29/2021, 12:12 PMso you can do something called the partial application of a node
but I want to be clear you're not using Kedro in the way we encourage you to do it
I would look at using hooks as you can access the catalog dynamically there
rather than trying to build a pipeline
d
Dhaval
12/29/2021, 12:14 PMOkay 😔 . I'm trying to take inspiration from the modular-spaceflights code and I'm trying to write in that way
Okay
d
datajoely
12/29/2021, 12:15 PMNo worries! It's a tricky one, we really encourage your inputs and outputs to live in Python
d
Dhaval
12/29/2021, 12:15 PMBut yeah, thanks for your help @User 😁
d
datajoely
12/29/2021, 12:15 PMthere is a subtle distinction between what should be static and what is configuration
there reason we're so annoying about this is that we believe it's a lot more maintianable and readable in the long run
d
Dhaval
12/29/2021, 2:49 PM@User I can't really understand how to go about this problem. I want to just write a list of tables that I want to extract from the SQL database and then run a profiler to save the html profiles and the data in PKL format inside the data/02_intermediate folder. I am attaching the flowchart for reference. If you could help me with this it would set up a big foundation for the project that I am currently working on. I really want this to be a modular pipeline based on the project structure and I have no proper experience. I'd really appreciate if you could help me with this
All i want to do is load the list of tables to be saved from the sql server and then save them with their filename which I get from the yml file. That's it😅
d
datajoely
12/29/2021, 5:36 PMOkay @User
Step 1. Update
create_descriptivesprofile.to_html()strpd.DataFrameStep 2. Wrap that in a regular
PipelineStep 3. Load from configuration the catalog entries you want to profile (somewhere in your
create_pipeline()Step 4. Loop over the results of step 3 and modular instances of Step 2 using
pipeline()You're already doing step 4 correctly, I think your current issues are in the actual node you've defined in your existing step 1 and 2
One question is the PKL file the SQL data or the JSON profiling data?
@User I actually sat down and had a go fixing your project
here is the PR
I'd say that the
after_catalog_createdj
j c h a r l e s
12/29/2021, 8:10 PM@Dhaval @datajoely thank you for having this discussion. Very helpful
d
Dhaval
12/29/2021, 8:21 PM@datajoely Thanks a lot for this. It's 2am here as of now, I'll go through this in the morning tomorrow . Also, another thing, i was able to achieve this on my own too, I'll share the nodes.py and pipeline.py file with you as well. Based on your inputs for best practices and the way Kedro works I've found a different solution. Let me know if that's works too. And again, thanks for your help @datajoely
j
j c h a r l e s
12/29/2021, 9:11 PMThis would be an awesome example to share via the docs
d
datajoely
12/29/2021, 10:18 PMYeah I’m in two minds if the hook implementation is good practice or not
On one hand it’s neat, but it’s less explicit than we usually like
d
Dhaval
12/30/2021, 7:49 AM@User I have added the PR here for my version here: https://github.com/DhavalThkkar/test_project/pull/2
I've gone through your implementation and it is pretty neat. Got to learn a lot about the Hook functionality based on your code. I'd really appreciate if you could go through my PR and share your feedback
Also, I have highlighted one weird error that I faced in the
pipelines.pyu
user
01/04/2022, 2:35 AM@User I took another approach to dynamic pipelines that I've found useful. Adding catalog entries via hooks added a bit of friction to my workflow and I've found adding the following to be more user-friendly:
1. Create a task that runs before any pipeline. The task runs a file called
create-catalog.pymessage has been deleted
d
Dhaval
01/04/2022, 7:06 AM@User Wouldn't you need to reinitialise the catalog files? I am guessing all of the catalog entries get initialised while
kedro runu
user
01/04/2022, 7:17 AMExactly
You make a preLaunchTask that updates the catalog
preLaunchTask is attached with kedro run
Which means it runs your catalog builder task before kedro run starts
which means that when kedro run happens, it uses the updated catalog files. the updated catalog files all exist on your machine before kedro run even starts basically
d
Dhaval
01/04/2022, 8:00 AM@User ,can you share a repo with this use case, it'll be easier for me to replicate
u
user
01/04/2022, 8:01 AMI am unable to share my repo directly but happy to answer more questions
d
Dhaval
01/04/2022, 8:11 AMWhere do I create these files. I am unable to understand that
u
user
01/04/2022, 8:13 AMAre you using Visual Studio Code?
If so, then you can open tasks and launch using Command + Shift + P then typing what you need
tasks
launch
Which files are you referring to when you asked "Where do I create these files?"
d
Dhaval
01/04/2022, 8:18 AMNow see, as per what you have mentioned, you said that you can create these catalog entries on the fly while the pipelines are being run. As of now, I am doing this in the nodes.py file of a pipeline
def create_descriptives(
data: pd.DataFrame,
parameters: Dict
):
file_name = parameters["file_name"]
# Create path for saving descriptives
path = create_folder(file_type="data", folder = "descriptives")
# Create the profiling report
profile = ProfileReport(
data, title=f"{file_name} Profiling Report",
config_file="config_minimal.yml"
)
# Save the report as an HTML file and JSON for further usage
profile.to_file(f"{path}/{file_name}.html")
json_data = json.loads(profile.to_json())
with open(f"./data/02_intermediate/{file_name}.json", "w") as file:
json.dump(json_data, file)
data.to_pickle(f"./data/02_intermediate/{file_name}.pkl")
return json_data, file_nameThe other approach that datajoely gave was of hooks but it wasn't that useable for my usecase. Your idea seems to go inline with what I am trying to achieve but I just can't understand the flow of things that you've mentioned. Hence, I asked for an example 😅
u
user
01/04/2022, 8:20 AMI did NOT say that "you can create these catalog entries on the fly while the pipelines are being run". I said that you can run a script before every run that updates your catalog file.
The caps on NOT is for emphasis, not meaning to be shouting etc
d
Dhaval
01/04/2022, 8:21 AMOh, okay. Got it 😆
u
user
01/04/2022, 8:22 AMSo what are you looping through, file_names?
d
Dhaval
01/04/2022, 8:23 AMYes, it is a list of tables that are present on the database
u
user
01/04/2022, 8:23 AMBasically what I do is, I save the list of the things that I need to loop through in a file in conf/base/parameters/tables.yml
Let's say you have a list of tables called: [users, events, activity]
I would create a script that loops through
conf/base/parameters/tables.ymlat the bottom of `catalog.yml`:
I would generate entries like this
#################### AUTOGENERATED #################
table_profile_results_for_users:
filepath: data/02_intermediate/users.json
type: json.JSONDataSet
table_profile_pickle_for_users:
filepath: data/02_intermediate/users.pkl
type: python.PickleDataSet
table_profile_results_for_events:
filepath: data/02_intermediate/events.json
type: json.JSONDataSet
table_profile_pickle_for_events:
filepath: data/02_intermediate/events.pkl
type: python.PickleDataSet
table_profile_results_for_activity:
filepath: data/02_intermediate/activity.json
type: json.JSONDataSet
table_profile_pickle_for_activity:
filepath: data/02_intermediate/activity.pkl
type: python.PickleDataSetthen in my create_pipeline i would loop through the same
conf/base/parameters/tables.ymlfrom kedro.pipeline import node
from kedro.pipeline.modular_pipeline import pipeline
import pandas as pd
def create_descriptives(
table_data: pd.DataFrame,
):
# Create the profiling report
profile = ProfileReport(table_data)
return profile.to_json(), profile.to_json().to_pickle()
def create_pipeline():
##
# Loop through users, events, activity
db_tables = [] # fill this by looping through your parameters file
pipeline_nodes = []
for table in db_tables:
pipeline_nodes += [
node(
func=create_descriptives,
inputs={"data": f"data_for_db_table_{table}"},
outputs=[
f"table_profile_results_for_{table}",
f"table_profile_pickle_for_{table}",
],
)
]
return pipeline(sum(pipeline_nodes))I'm not exactly sure what inputs you're using as the input for this function
but this is essentially what I have been doing
I have a script that dynamically adds to the end of my catalog.yml
and then can create the nodes based on my parameters file