rafael.gildin
09/27/2022, 2:54 PMVici
09/27/2022, 3:12 PMuser
10/03/2022, 1:34 PMBarros
10/03/2022, 3:13 PMBarros
10/03/2022, 3:14 PMSeth
10/04/2022, 12:11 PMnoklam
10/04/2022, 12:21 PMSeth
10/04/2022, 12:29 PMnoklam
10/04/2022, 12:32 PMSeth
10/04/2022, 12:33 PMantheas
10/05/2022, 3:36 PMuser
10/07/2022, 8:07 AMuser
10/07/2022, 2:04 PMuser
10/07/2022, 4:46 PMuser
10/08/2022, 6:36 PMwilliamc
10/10/2022, 2:31 AMkpro
and I've abstracted a bunch of stuff into kpro.coollib
. If I package it up with kedro package
and distribute it, do I just need to do import kpro.coollib
on my destination machine to use it? In this case I'm not interested in running my pipelines, just the library code. Thanks!noklam
10/10/2022, 8:04 AMnickolas da rocha machado
10/10/2022, 2:47 PMpython
# pipeline_registry.py
pipelines["partitioned"] = Pipeline([node(print, 'partitioned', None)])
yml
# catalog.yml
partitioned:
type: PartitionedDataSet
dataset: pandas.CSVDataSet
path: abfs://...dfs.core.windows/...
credentials: lab
filename_suffix: .csv
log
[10/10/22 14:39:54] INFO Kedro project
[10/10/22 14:39:55] INFO Loading data from 'partitioned' (PartitionedDataSet)...
noklam
10/10/2022, 2:49 PMnickolas da rocha machado
10/10/2022, 2:50 PMnickolas da rocha machado
10/10/2022, 2:51 PMnickolas da rocha machado
10/10/2022, 2:53 PMPartitionedDataSet._list_partitions
. it doesn't look like a Kedro problem, but an adlfs problem, since it only stops execution when reaching its glob method.nickolas da rocha machado
10/10/2022, 3:05 PMOnéira
10/11/2022, 8:55 AMkedro build-docs
etc. will be deprecated in 0.19. Are there other commands which are planned to replace them? Or a new procedure to build the doc or requirements?Merel
10/11/2022, 10:21 AMkedro build-docs
specifically we have a follow up task to provide a good alternative: https://github.com/kedro-org/kedro/issues/1618 Another thought we have for the commands is to move them to a community supported plugin. Please leave your thoughts on https://github.com/kedro-org/kedro/issues/1622 if this sounds useful to you 🙂Onéira
10/11/2022, 1:41 PMfarisology
10/12/2022, 7:10 AMkedro run
in a local machine and it starts executing it says running kedro session start
does this mean it runs a local server?
2- If we are running Kedro DAG will that mean the kedro will start a local server in our airflow? what ports will it use?
3- The kedro-airflow has been archived, what does this mean for the support of the project?
4- Since there is limited documentation on running kedro in stand-alone airflow, does this mean kedro airflow will only be a sustainable approach with astronomer going forward?
5- why there is kedro operator instead of python or base operator?Merel
10/12/2022, 9:36 AMkedro-airflow
plugin for this?
3. You can definitely run Kedro on stand-alone airflow as well. The Kedro maintainer team tries to keep documentation and deployment guides up to date, but we don't always manage to write about all possible ways of deploying Kedro projects. Let us know where we can help here!
4. I don't really understand your question, is this related to airflow/`kedro-airflow` again?lancechua
10/12/2022, 11:42 AMGenericDataSet
class factory make sense?
I'm looking to write a class factory with the following signature:
python
def create_generic_dataset(
name: str,
load: Callable[[Path, ...], pandas.DataFrame],
save: Callable[[pandas.DataFrame, Path, ...], None]
):
...
or is subclassing the AbstractaDataSet
still the preferred method?datajoely
10/12/2022, 11:43 AM