Does the new catalog lazy loading somehow hit the ...
# beginners-need-helpw
WolVez
09/23/2021, 5:18 PMDoes the new catalog lazy loading somehow hit the describe functionality of a dataset now, prior to the dataset being initialized?
WolVez
09/23/2021, 5:19 PMI have the following custom dataset:
d
datajoely
09/23/2021, 5:21 PMI’m not entirely sure what you’re looking to achieve
datajoely
09/23/2021, 5:22 PMAlso you can use the regular SQL datasets in Kedro with snowflake as long as you install https://pypi.org/project/snowflake-sqlalchemy/
w
WolVez
09/23/2021, 5:22 PMya still trying to type everything out
WolVez
09/23/2021, 5:22 PMthe issue is the _describe function is being hit during the initialization prior to the self.sql = sql
d
datajoely
09/23/2021, 5:22 PMAnd is the problem speed or something
w
WolVez
09/23/2021, 5:23 PMNo its that the run fails due to catalog load error because the dataset cannot be instantiated.
d
datajoely
09/23/2021, 5:23 PMOh got you
datajoely
09/23/2021, 5:23 PMI mean you can return an empty dictionary in the describe method
datajoely
09/23/2021, 5:24 PMBut I would encourage you to use the regular pandas.SqlQueryDataSet with the snowflake engine library as it’s tested
w
WolVez
09/23/2021, 5:33 PM@User we basically are doing what you are saying, but just managing credentials inside of a singleton in another package. Reguardless, I am baffeled as to why _describe is being hit prior to __init__ or even how that works.
d
datajoely
09/23/2021, 5:37 PMYeah it’s happening somewhere as the catalog is built up
w
WolVez
09/23/2021, 5:37 PMAlso, note this worked until upgrading from 17.2 to 17.5.
d
datajoely
09/23/2021, 5:37 PMI need to finish for the day - but I will look into this first thing
w
WolVez
09/23/2021, 5:37 PMSounds good! Thanks for the help @User!
WolVez
09/23/2021, 5:49 PMI found the problem. Super() is hitting the AbstractDataSet class which now has the following definition for
__str__WolVez
09/23/2021, 5:51 PMThis seems a bit dangerous to me, given that the AbstractDataset is kinda designed to be inherited in the same way my dataset is created. But anyone including the input params in the same way that I did (super first then self.x being created) will hit this same error.
l
Lorena
09/24/2021, 9:29 AMHi @User ! AbstractDataSet doesn't have a constructor, so when extending it you don't need to call
super().__init__SnowflakeAbstractDataSetstr()Lorena
09/24/2021, 9:31 AMOr I'd also make use of
mro()w
WolVez
09/24/2021, 4:55 PM@User, I am not entirely following what you are saying.
From what I can tell, to have a custom datasets, it has to be an inheritance from the abstract class such that when the catalog is built and passed to the AbstractDataset it is able to build, if you remove the AbstractDataset from the inheritance, it no longer functions..
__str__ is called like __init__ during the instantiation of the class. So what is happening is that in my class is defining _describe with an eliment which is not defined yet, as it is defined after the super(). Simply moving super() down resolves it.
I suppose a viable solution would be to focus super() the Snowflake class, but again I suspect a lot of people will run into this same issue with how it is currently setup.
d
datajoely
09/27/2021, 8:23 AMCan we see the
Snowflakew
WolVez
09/27/2021, 2:22 PM@User There are more to these classes, but I have shorten them to what you would care about.
d
datajoely
09/27/2021, 2:30 PMThanks @User we will get back to you shortly
2 Views