hey guys I'm facing some issues when trying to write data from one BQ table to another, I did set the if_exists: 'append' but when I do it throws error

Can you show us your catalog entry? To may have to put it under your save_args

I have two pipelines for: monthlypipeline = daily_data_pipeline + monthlyaggregate_pipe weeklypipeline = daily_data_pipeline +weeklyaggregate_pipe Since the pipelines share nodes, i get error. what is the best way to write this in kedro?

What kind of error are you getting? These kinds of dependencies are what Kedro is designed for, it should resolves the DAGs and merged any repeated nodes automatically.

i get error for "Pipeline nodes must have unique names". what is the best way to handle this? any sample code

Provide a variation of the

name

arg, or actually drop the explicit name altogether

sorry answering you just now, I did check the save args and now its working, I just miss place a args...

Hi, i would like to ask you something guys ... i have a question regarding the execution of the nodes. nodes that come from the same dataset are executed one after the other as if they were in a serial pipeline?, if so, is there a way to open a thread for each process to make it go faster? Here i attach some pictures to have a reference of what i am talking about

Look for runner, by default it uses a sequentialrunner, there are parallelrunner and threadrunner as well.

so threadrunner would handle this case?. If i use Sequential runner, the way the nodes are executed are like in the second image?

It depends on what you are doing, threading isn't straight forward in Python due to GIl

It will be executed one by one, but not guaranteed a, b, c.

In a DAG they are independent, so it shouldn't matter who executed first.

Ok thanks i will take a look at it 😄

Hi, I am using Kedro for my ML project. I am happy that I got to know Kedro, because it makes it much easier to implement a ML model as a data scientist. Currently I am working to make Kedro ML project run on Spark cluster that uses Hive database. For this I make use of SparkHiveDataset of Kedro. The problem I faced was that there was a mismatch between variable name used in the codes of Kedro and the variable name used in Hive database that I was using. I use Kedro version 0.17.7 and below is the code of function _exists() in spark_hive_dataset.py. def _exists(self) -> bool: if ( self._get_spark() .sql("show databases") .filter(col("namespace") == lit(self._database)) .take(1) ): self._get_spark().sql(f"use {self._database}") if ( self._get_spark() .sql("show tables") .filter(col("tableName") == lit(self._table)) .take(1) ): return True return False I have changed "namespace" to "databaseName" and after that I was able to load the data from table of Hive database. What I found in the newest version 0.18.2 is that the code of function _exists() is replaced with new code that make use of Spark function to do the same. def _exists(self) -> bool: # noqa # pylint:disable=protected-access return ( self._get_spark() ._jsparkSession.catalog() .tableExists(self._database, self._table) ) I had trouble (getting various errors) to use version 0.18.2, so I kept the version that I am using (0.17.7) and only changed the codes in function _exists() in spark_hive_dataset.py with the updated codes of version 0.18.2. I would like to ask whether it is ok to use it like this. Will there be no problem/issue in using it like this?

Did you just overwrite the source code? It's fine if you are just doing it yourself. 1. Ideally if you can just upgrade to

0.18.2

, you can share the errors that you get and we can try work together and help you with the migration. 2. If upgrade is not possible, a slightly better option is make your own

CustomHiveDataSet

, that just inherit from the original

SparkHiveDataSet

and you implement the

_exists

method to override its behavior, you can package that in your project. One addition thing you need to is update the

DataSet

type in your

catalog,yml

to point to your custom implementation instead.

I have overwritten the source code with updated source code. It is good to know that this is in principal not a problem. I plan to do the migration to the newer version in the future, for which I have to reserve some time for it. I will make CustomHiveDataset to make it work better if I have enough time to do it 🙂 . Thank you for your help, Kedro team!

Just be aware that it will only work on your own laptop, once you need to share/distribute the code it won't work anymore.

This Kedro project is only placed in a platform. Other users will use this Kedro project on the platform by providing their data and get the predictions back. So I think sharing of code will not take place.

In that sense, it's kind of a deployed project, personally, I would be very cautious to do this in a production environment. It's dangerous since once you re-deploy or any CI/CD reinstall the dependencies things will break and no one will know why.

Thank you for the information! Then I will create CustomHiveDataset to do the work. Will this be sufficient enough for deployed project?

I see that it is relatively simple to create a custom dataset. I just copied all the contents of SparkHiveDataSet and placed them in CustomHiveDataSet. I only changed the codes in _exists() function. By doing this I am now able to run the project the same way as I did before. Is this sufficient for deployment project or has Kedro to be migrated to the newest version?

Hi All. I wanted to test experiment tracking using kedro. I did not want to use the mlflow plugin and just wanted to try out what Kedro can do. I was using the space flights pipeline and defined a tracking.JSONDataSet in the data catalog and modified the pipeline to output the metrics. However, I do not see that metrics dataset being written to disk. It gives below message```INFO Model has a coefficient R^2 of 0.449 on test data. nodes.py:56 INFO Saving data to 'data_science.candidate_modelling_pipeline.metrics' (MemoryDataSet)... ```

I also tried to hard-code the metrics dataset with a sample json file just to see how kedro viz will show the metric value, however, kedro viz does not show the value. It just shows the metrics dataset as an output on the viz

it's likely because you are using namespace piepline, your dataset name need to be "data_science.candidate_modelling_pipeline.metrics" base on your log

Yes, I am using a namespace pipeline, let me change that and see what happens.

that solved the problem and I am getting the metrics and my kedro viz can see the metric values. How do I get the experiments on my kedro viz? It still says "You don't have any experiments"

Did you update your settings.py to use the SQLiteStote?

Do you mean instructions on this page? https://kedro.readthedocs.io/en/latest/tutorial/set_up_experiment_tracking.html I just found it, have to read it. I was reading the stable docs and hence did not find this page.

https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html#set-up-the-session-store This section