WolVez
10/08/2021, 7:07 PM--async
is implemented with _run_node_async
, there doesn't seem to be a good way to achieve this that I can think of, other than making my own dataset which applies an async get while running in non async mode (as stacking async on async is bad). However, this must be a common use case you have overcome?
I feel like adding a hook to append to the node would be a good thing here.datajoely
10/08/2021, 7:45 PMWolVez
10/08/2021, 7:51 PMdatajoely
10/09/2021, 10:52 AMWolVez
10/11/2021, 1:34 AM_run_node_async
to change from a list of node items to an extended list of all datasets (to include partitioned datasets) might be nice speed improvement at somepoint. Just a thought. I added my own dataset for my use case to manage it.datajoely
10/11/2021, 6:52 AM_run_node_async
we could maybe discuss the approach in detail there?wulfcrona
10/11/2021, 8:49 AMdatajoely
10/11/2021, 9:19 AMdmb23
10/13/2021, 8:34 AMdata/01_raw/my_metadata.db
, but specify this location in the credentials.yml (which is not the first place I would look if future me has to change something). I think it might also be possible to put it in a globals.yml with a TemplatedConfigLoader, but I did not find a way to include it in a catalog.yml where I would search for it when I want to define SQLQueryDataSet(s)
- and in that context: the documentation states that it might be possible (at least for a SQLTableDataSet) to provide the connection string not in credentials, but in the load_args. But both classes SQLTableDataSet and SQLQueryDataSet have a check for credentials[con] that raises a DataSetError when it is not found. Is there some way around that which I am missing? (or should I offer to change the documentation accordingly?)datajoely
10/13/2021, 9:00 AMdatajoely
10/13/2021, 1:33 PMBastian
10/13/2021, 2:37 PMkedro pipeline pull
copies the source code into the project. I am afraid that this would lead to the different usages of the modular pipeline diverging over time. What is the suggested way of reusing modular pipelines in different projects?datajoely
10/13/2021, 6:05 PMBastian
10/14/2021, 8:02 AMuser
10/15/2021, 5:41 PMdatajoely
10/15/2021, 5:42 PMuser
10/15/2021, 5:42 PMdatajoely
10/15/2021, 5:42 PMuser
10/15/2021, 5:43 PMdatajoely
10/15/2021, 5:43 PMuser
10/15/2021, 5:43 PMdatajoely
10/15/2021, 5:43 PMdatajoely
10/15/2021, 5:44 PMuser
10/15/2021, 5:47 PMuser
10/15/2021, 5:47 PMdatajoely
10/15/2021, 6:35 PMuser
10/15/2021, 6:38 PMdatajoely
10/15/2021, 6:39 PMbgereke
10/15/2021, 11:21 PMdatajoely
10/16/2021, 7:07 AM