Dhaval
12/31/2021, 5:06 PMDhaval
01/03/2022, 5:12 AMGalileo-Galilei
01/03/2022, 9:10 AMGalileo-Galilei
01/03/2022, 8:57 PMDhaval
01/04/2022, 6:54 AMGalileo-Galilei
01/04/2022, 11:55 AMpython
mlflow.autolog()
for hyperparams in hyperparams_candidates:
with mlflow.start_run(nested=True):
model.train(**hyperparams)
which will create sub runs inside main one. Is it what you want?Dhaval
01/05/2022, 3:40 PMkedro mlflow init
and then kedro run
on the repository mentioned above. The thing that is happening as of now is, the parameters aren't logged separately as separate runs. Its all present in just one single run. Since the models are different I want the runs recorded separately. How do I achieve that?
Do note that the modelling pipeline in this repo has used namespace for grouping multiple regression modelsGalileo-Galilei
01/05/2022, 9:27 PMGalileo-Galilei
01/05/2022, 9:28 PMpython
# inside a custom AbstractRunner
with mlflow.start_run(nested=True):
runner.run(pipeline, catalog)
- do all hyperparameter tuning inside a node, and laucnh sub run like described above:
python
# inside a node
for hyperparams in hyperparams_candidates:
with mlflow.start_run(nested=True):
model.train(**hyperparams)
There is an interesting discussion on the same theme here: https://github.com/Galileo-Galilei/kedro-mlflow/issues/246ChainYo
01/28/2022, 2:44 PMCelery
to run multiple Kedro
pipelines on a daily basis ?Dhaval
02/03/2022, 8:01 AM'postgresql://postgres:postgres@localhost:5432/mlflow_db'
When I use this command:
model = mlflow.pyfunc.load_model(
model_uri=f"models:/temp/1"
)
I get the following error:
Note: temp here is the model name and 1 is the version of the registered modelGalileo-Galilei
02/04/2022, 12:21 PMGalileo-Galilei
02/04/2022, 12:24 PMfrom kedro.framework.session import KedroSession
from kedro_mlflow.config import get_mlflow_config
with KedroSession.create() as session:
mlflow_config=get_mlflow_config()
mlflow_config.setup()
< your code>
Dhaval
02/08/2022, 1:11 PMlbonini
02/10/2022, 5:15 PMkedro build-docs
? Nodes, pipelines and subpackages in general (docstrings) are not showing in docsantony.milne
02/10/2022, 5:25 PMkedro build-docs
- no need to modify index.rst. Something that's tripped me up before: you need __init__.py
files everywhere for Sphinx to find all the right moduleslbonini
02/10/2022, 5:50 PMkedro new --starter=pandas-iris
, there are __init__.py
files in every folder. đIS_0102
02/10/2022, 8:31 PMdatajoely
02/10/2022, 9:08 PMArnaldo
02/11/2022, 12:26 PMoutput_metric_name_defined_in_node_output:
type: kedro_mlflow.io.metrics.MlflowMetricsDataSet
> Where can I find the run-ids on the web ui?
If I understood it correctly, the run id is the top of the run page (see image attached)Arnaldo
02/11/2022, 12:28 PMlogs/info.log
. Therefore, probably you could call mlflow.log_artifact("logs/info.log")
to save itantony.milne
02/11/2022, 4:08 PMIS_0102
02/11/2022, 10:48 PMDhaval
02/12/2022, 7:05 AMGalileo-Galilei
02/12/2022, 7:36 AMGalileo-Galilei
02/12/2022, 7:46 AMIS_0102
02/13/2022, 5:22 PMArnaldo
02/14/2022, 12:32 PMmlflow==1.23.1
as well, but with kedro-mlflow==0.7.6
Arnaldo
02/14/2022, 12:41 PMlog/info.log
, you can change the info_file_handler
in conf/<env>/logging.yml
to the following:
info_file_handler:
class: logging.FileHandler
level: INFO
formatter: simple
filename: logs/info.log
mode: w
encoding: utf8
delay: True
Dhaval
02/14/2022, 4:16 PMhttp://192.168.0.150:9000
with the bucket name mlflow
The minio server is running locally so I've added the following entries in the credentials.yml
file of my kedro project:
mlflow_creds:
MLFLOW_S3_ENDPOINT_URL: 'http://192.168.0.150:9000'
AWS_ACCESS_KEY_ID: 'minioadmin'
AWS_SECRET_ACCESS_KEY: 'minioadmin'
These are the values for the mlflow.yml file:
server:
mlflow_tracking_uri: 'postgresql://postgres:postgres@localhost:5432/mlflow_db'
stores_environment_variables: {}
credentials: mlflow_creds
What I want to do is set postgres to track metrics and manage registered models and use minio's bucket(mlflow
) to save the artifacts .
The above configuration saves all the runs locally inside ./mlruns
folder, I want it to point to the s3://mlflow
bucket. Please helpDhaval
02/14/2022, 4:16 PMhttp://192.168.0.150:9000
with the bucket name mlflow
The minio server is running locally so I've added the following entries in the credentials.yml
file of my kedro project:
mlflow_creds:
MLFLOW_S3_ENDPOINT_URL: 'http://192.168.0.150:9000'
AWS_ACCESS_KEY_ID: 'minioadmin'
AWS_SECRET_ACCESS_KEY: 'minioadmin'
These are the values for the mlflow.yml file:
server:
mlflow_tracking_uri: 'postgresql://postgres:postgres@localhost:5432/mlflow_db'
stores_environment_variables: {}
credentials: mlflow_creds
What I want to do is set postgres to track metrics and manage registered models and use minio's bucket(mlflow
) to save the artifacts .
The above configuration saves all the runs locally inside ./mlruns
folder, I want it to point to the s3://mlflow
bucket. Please help