https://kedro.org/ logo
Join Discord
Channels
advanced-need-help
announcements
beginners-need-help
introductions
job-posting
plugins-integrations
random
resources
welcome
Powered by Linen
Title
c

ChainYo

01/13/2022, 10:16 AM
Is there a way to automatically add a dataset to catalog based on params ? Because the API endpoint is exactly the same but the name
d

datajoely

01/13/2022, 10:50 AM
Hey @User started a thread so we don't pollute the main channel
c

ChainYo

01/13/2022, 10:51 AM
This is what I have got for the moment
message has been deleted
message has been deleted
message has been deleted
that's fine but I wondered if I can get it compact 😄
d

datajoely

01/13/2022, 10:54 AM
python
class APICatalogHooks:

    @hook_impl
    def after_catalog_created(self, catalog, conf_catalog, conf_creds, feed_dict, save_version, load_versions, run_id):
        """This is an advanced use of the catalog hook so that we create the right
        catalog entries at runtime based on the inputs to the `params:api_stuff`. The alternative
        to this is that your non-technical users would have to create the three output
        dataset entries in the catalog for every input they declare
        """

        api_datasets_to_create = feed_dict['params:api_stuff']

        for dataset in api_datasets_to_create:
            new_entry = {
                f"{dataset['compare_1']}_vs_{dataset['compare_2']}_market_chart" : APIDataSet(url=f'https://api.coingecko.com...{params}...'),
            }
            catalog.add(new_entry, replace=True)
you then add this hook to 
settings.py
and it will dynamically create the necessary catalog entries
long catalog name makes i super ugly here
but hopefully you get the point
c

ChainYo

01/13/2022, 10:58 AM
long catalog names are okay because I need this kind of info to be accurate (because it could be vs usd, vs eur, vs gbp... and it could be anything else than market chart)
message has been deleted
btw thanks for the code snippets
I will try it now
d

datajoely

01/13/2022, 10:59 AM
Good luck!
to explain how this works
by default Kedro will create 
MemoryDataSets
for every input/output in the pipeline definition NOT in the catalog
c

ChainYo

01/13/2022, 11:00 AM
I create my hook custom class then I add it to 
settings.py
so when I run the kedro project it will do the job
yes I got this point in mind
d

datajoely

01/13/2022, 11:01 AM
then this hook will replace the 
MemoryDataSets
with the real references to the 
APIDataSet
instances
c

ChainYo

01/13/2022, 11:02 AM
Perfect this was exactly what I needed
thanks a lot
d

datajoely

01/13/2022, 11:02 AM
💪
c

ChainYo

01/13/2022, 11:04 AM
Are these arguments optionnal or mandatory ? 
(self, catalog, conf_catalog, conf_creds, feed_dict, save_version, load_versions, run_id)
oh ok I saw on the template project hooks that they are also here
d

datajoely

01/13/2022, 11:05 AM
they are mandatory
but you don't have to use them
https://kedro.readthedocs.io/en/latest/07_extend_kedro/02_hooks.html
you can read about how they work here
but essentially each of the hooks available like 
before_node_run
has a function signature available that exposes certain things - in this case the catlaog
c

ChainYo

01/13/2022, 11:23 AM
you are using this 
feed_dict['params:api_stuff']
If my comprehension is correct, I create a file in 
~/conf/base/feed_dict.yml
with this inside : 
yml
params:
  - bitcoin
  - ethereum
  - chiliz
...
d

datajoely

01/13/2022, 11:23 AM
so if your 
parameters.yaml
looks like
crypto_currencies:
   - bitcoin
   - ethereum
   - chiliz
you would have to do 
feed_dict['params:crypto_currencies']
and you would get a python list of the 3 strings in that key
c

ChainYo

01/13/2022, 11:25 AM
ok I see
what is the import for feed_dict?
d

datajoely

01/13/2022, 11:25 AM
then I would recommend using itertools to generate the combinations in your hook
the 
feed_dict
is a bit of an old name
c

ChainYo

01/13/2022, 11:26 AM
add_feed_dict()
right now ?
d

datajoely

01/13/2022, 11:26 AM
it's for advanced usage to inject more paramters
but its a part of the library we only expose if you're doing advanced things like hooks
c

ChainYo

01/13/2022, 11:30 AM
Sorry I didn't understand the last part 😄
Never used itertools, I do list comprehension in python or dict comprehension
d

datajoely

01/13/2022, 11:32 AM
message has been deleted
if you give 
combinations
a interable and the number 2 it will generate every pair
without doing each pair in reverse too
combinations
returns a generator
so you need to iterate through it to get the pairs out
list
does that for readability here, but you could do a for loop too
itertools
is genuinely one of the best bits of the standard library
if you want the reverse pairs it's got that sorted with 
permutations
message has been deleted
c

ChainYo

01/13/2022, 11:34 AM
would be usefull when I implement eur or gbp aside usd
nice thanks
so helpfull !
d

datajoely

01/13/2022, 11:35 AM
my pleasure
https://docs.python.org/3/library/itertools.html
c

ChainYo

01/13/2022, 1:03 PM
I read the config part about config file and loading, but it's not super clear about using config params inside hook
I know how to use Hydra config manager, so maybe it's another paradigm for kedro but I can't understand how to use them
Ok I found ressource that will help with parameters comprehension 🙂
https://waylonwalker.com/kedro-parameters/
d

datajoely

01/13/2022, 1:11 PM
The params appear in the 
feed_dict
kwarg
which you can use
honestly the easiest technique is to put a 
breakpoint()
in the hook body
and then inspect it at runtime to get a sense of what's avaialble
the parameters in the article linked are the standard way of doing it - you are doing an advanced mechanism for auto-generating catalog entries
c

ChainYo

01/13/2022, 1:13 PM
oh cmon it's my fault, I didn't add 
feed_dict
to my function args
That's why I couldn't find feed_dict...
d

datajoely

01/13/2022, 1:14 PM
no worries
what I do
c

ChainYo

01/13/2022, 1:14 PM
it seems that the doc is missing 
feed_dict
btw
https://kedro.readthedocs.io/en/latest/07_extend_kedro/02_hooks.html#hook-implementation
d

datajoely

01/13/2022, 1:14 PM
ah good spot!
what I do is copy the arguments from this page
https://kedro.readthedocs.io/en/stable/_modules/kedro/framework/hooks/specs.html#NodeSpecs
c

ChainYo

01/13/2022, 3:43 PM
the first code snippets you provided seems to be broken at the 
catalog.add()
line but fixed it with the source code
python
            catalog.add(
                data_set_name=f"{dataset[0]}_vs_{dataset[1]}_market_chart",
                data_set=APIDataSet(
                    url=f"https://api.coingecko.com/api/v3/coins/{dataset[0]}/market_chart?vs_currency={dataset[1]}&days=max&interval=daily"
                ),
                replace=True
            )
the 
new_entry
dict doesn't work the 
kedro run pipeline
raise an error because it's missing a name for the dataset, so I removed 
new_entry
and added both inputs directly into the 
add()
function 🙂
But that works, thanks again.
d

datajoely

01/13/2022, 4:02 PM
amazing!
u

user

01/13/2022, 4:22 PM
Not sure if this is helpful, but I have a "pre launch pipeline" that builds csv files from a database, that runs before my pipelines. Then after this pipeline gets these CSVs, I have an "autogenerated" section for my parameters files and for my catalog.yml. I have a script that goes through and updates both of these files to include the values from the csv from my prelaunch pipeline. Once both my parameters and catalog are updated with these entries, in my create_pipeline I look through the keys in my parameters files in order to create the nodes of my pipeline. I've found when I tried to use the catalog.add, it was easy to get confused and also that there was some case (that I cant remember now), that writing updates to the params & catalog file fixed.
c

ChainYo

01/13/2022, 4:46 PM
I'm curious about the 
in order to create the nodes of my pipeline
👀
Do you define a template node to use for generating N nodes based on that template ?
d

datajoely

01/13/2022, 4:57 PM
I'll say that this isn't entirely endorsed by the Kedro core team - the only reason being we have observed the more dynamic your pipeline definition the harder it is to debug or onboard new team members
obviously every situation is different and it's great to see the creative ways people use the tool
c

ChainYo

01/13/2022, 5:26 PM
I get it
I will probably go by adding nodes manually for now 🙂
Even if automatic datasets add is useless if I can't get automatic nodes add 😄
u

user

01/14/2022, 1:48 AM
yeah I automatically generate a parameters file, which is then parsed in order to generate pipeline nodes
c

ChainYo

01/15/2022, 5:04 PM
I'm on another path
I will make a pipeline and give params at running
via 
kedro run --params key:value key2:value ...
so I can define a standard pipeline without worrying about datasets/nodes that depends on the params giving at runtime
I think it will be easier to maintain and to understand
So I use for example 
kedro run --params currency:bitcoin,compare:usd
How do I format the node inputs ? I have tried : 
python
node(
  func=format_market_chart_to_dataframe,
  inputs="params:currency_vs_params:compare_market_chart",
  outputs="fetched_params:currency_vs_params:compare_market_chart",
  name="fetched_data_node",
),
I tried also with f-strings but that doesn't work I get this error : 
bash
ValueError: Pipeline input(s) {'params:currency_vs_params:compare_market_chart'} not found in the DataCatalog
I also tried by adding single quote inside multi-quote
I will go with standard naming for datasets without variables, but I'm still curious if there is a way to add parameters to inputs (wihtout putting them in a list).
Sorry for spamming, but 
Kedro
is so awesome I want to use it at his full potential 😄
d

datajoely

01/16/2022, 4:11 PM
No worries
I'm not entirely sure what you're trying to do here
can you show me how you've formatted your paramters?
are you using any hooks?
c

ChainYo

01/17/2022, 7:01 PM
Yes I use the hook you gave me last time 
python
class APICatalogHooks:
    @hook_impl
    def after_catalog_created(
        self, 
        catalog: DataCatalog,
        conf_catalog: Dict[str, Any],
        conf_creds: Dict[str, Any],
        feed_dict: Dict[str, Any],
        save_version: str,
        load_versions: Dict[str, str],
        run_id: str,
    ) -> None:
        """
        This hook is called after the catalog is created. It creates one entry in the catalog per crypto currency
        listed in the config file.
        """
        currency = feed_dict["params:currency"]
        compare = feed_dict["params:compare"]

        catalog.add(
            data_set_name=f"inputs_market_chart",
            data_set=APIDataSet(
                url=f"https://api.coingecko.com/api/v3/coins/{currency}/market_chart?vs_currency={compare}&days=max&interval=daily"
            ),
            replace=True
        )
for running I add 2 parameters 
kedro run --params currency:bitcoin,compare:usd
With this way I can use the same pipelines by changing parameters I give by running them
And I decided to use standard name for datasets in my pipelines 
python
def create_pipeline(**kwargs):
    return Pipeline(
        [
            node(
                func=fetch_data_to_dataframe,
                inputs=["params:currency", "params:compare"],
                outputs="fetched_market_chart",
                name="fetching_data_node",
            ),
        ]
    )
d

datajoely

01/17/2022, 7:04 PM
nice so is it working?
what error are you getting
c

ChainYo

01/17/2022, 7:05 PM
no that's fine 🙂
d

datajoely

01/17/2022, 7:05 PM
💪
c

ChainYo

01/17/2022, 7:05 PM
I wanted to use dynamic name for datasets, but it's useless, it's easier like that
d

datajoely

01/17/2022, 7:06 PM
nice
View count: 3
Back to #beginners-need-helpJoin thread in Discord