deepyaman
06/16/2022, 4:25 AMkedro-mlflow
that I've looked into tonight (mostly MlflowModelSaverDataSet
) is really well-designed, and I appreciate it! Was going to go down the route of wrapping `save_model`/`load_model` and supporting flavors and everything, and so glad a search while looking into it turned up this specific dataset implementation, since it's much better than would have done myself. @Galileo-Galilei 🙂Galileo-Galilei
06/16/2022, 8:10 PMdatajoely
06/20/2022, 9:07 AMAnarpego
09/07/2022, 7:15 PMkedro kubeflow compile
any idea? pleasemarrrcin
09/08/2022, 11:05 AMAnarpego
09/08/2022, 2:18 PMdevintaylor03
09/12/2022, 9:41 AMdbx
could help out. When running a dbx execute
job we cannot get the kedro logs to stream to console, this seems to be an issue with dbx
rather than kedro but hoping someone on here has found a way around this! Link to issue for more context https://github.com/databrickslabs/dbx/issues/463. Thanks!Goss
09/12/2022, 7:01 PMkedro docker build
looks like it is running forever:
Step 7/12 : RUN groupadd -f -g ${KEDRO_GID} kedro_group && useradd -d /home/kedro -s /bin/bash -g ${KEDRO_GID} -u ${KEDRO_UID} kedro
---> Running in eacc1a27e787
But when I use docker ps -a
I notice that the container that is supposedly hanging during that build step actually shows that it already completed:
eacc1a27e787 c18ff7793d12 "/bin/sh -c 'groupad…" 2 minutes ago Exited (0) 2 minutes ago
Any ideas what is going on here?Goss
09/12/2022, 7:24 PMStep 7/12 : RUN groupadd -f -g ${KEDRO_GID} kedro_group && useradd -d /home/kedro -s /bin/bash -g ${KEDRO_GID} -u ${KEDRO_UID} kedro
---> Running in eacc1a27e787
Error processing tar file(exit status 1): write /var/log/lastlog: no space left on device
It's filling up over 300 GB of disk so something is not working right. Filing an issue on Github.Yetunde
09/13/2022, 2:30 PMGoss
09/14/2022, 3:28 PMem-pe
09/15/2022, 7:39 AMvolume
and extra_volumes
sections in config https://kedro-kubeflow.readthedocs.io/en/0.7.0/source/02_installation/02_configuration.htmlRjify
09/15/2022, 9:54 PMGoss
09/19/2022, 4:37 PMGoss
09/19/2022, 5:10 PMmlpipeline-minio-artifact
with accesskey
and secretkey
. So how can I map through the credentials stored in this secret so that the data catalog can use them?Goss
09/20/2022, 5:54 PMnoklam
09/20/2022, 5:58 PMnoklam
09/20/2022, 6:01 PMGoss
09/20/2022, 6:32 PMkfp.aws
functionality can be used to access a Minio S3 endpoint and that kfp.aws.use_aws_secret
map the components of the existing Minio secret through into typical AWS credential environment variables... so, I think this can actually work.noklam
09/20/2022, 7:01 PMGoss
09/21/2022, 4:49 PMGoss
09/21/2022, 9:24 PMGoss
09/26/2022, 2:47 PMlatest
as the image version. In theory, if imagePullPolicy
is Always, KFP should check the registry to see if the sha on the image has changed and repull the image when provisioning a new container. This is not happening like it should--I can see KFP using a new container (via a new containerId
) but the sha of the image used does not update to the latest in the repository. I have also applied Always
as the imagePullPolicy
for the node in question (even though I think that is already inherited). Still it doesn't work. I could switch to semantic versioning of the image (instead of latest
) BUT it seems like Kedro is designed to work with latest
and also that will lead to a build-up of images in the registry (at least in the development stage).
Does anyone know what might be going on here or how to get KFP to use the most recent image when using latest
versioning?Goss
10/05/2022, 5:02 PMGoss
10/05/2022, 5:41 PMbuild_kubeflow_pipeline.py
python file to compile the pipeline to a yaml file that is then uploaded. But the kedro-kubeflow plugin has a kedro kubeflow compile
function that uses conf/base/kubeflow.yaml
to generate the pipeline yaml. Is compiling from kubeflow.yaml
the correct way to go here?
2. I'm trying to get my pipeline to compile using kubeflow.yaml
but kedro kubeflow compile
seems to want to upload to the host:
too... which seems odd for a compile semantic. In my case, I don't have cluster access except through the UI... so how can I truly just have it only compile?Jose Alejandro M
10/06/2022, 5:32 PMAnarpego
10/12/2022, 2:37 AMAllen Ma
10/18/2022, 2:01 AMAllen Ma
10/18/2022, 2:40 AMYetunde
12/08/2022, 1:26 PM