I have more of a conceptual question, for my latest project one of the features are scraped from a react-web app. To do this I need a path to chrome driver in addition to some python libraries. What is the Kedro way to solve this? Should I just add them to the catalog and feed to the nodes for create a special folder and have the path in the parameters?
02/28/2022, 12:50 PM
I think this makes sense - a custom dataset could also work for this
Maybe post a version of what the YAML could look like and we can think through the problem
02/28/2022, 1:46 PM
After spending some time on it it made more sense to implement it as a custom dataset since I'd like to avoid hard coded paths. The custom dataset is more or less a copy of the example from the documentation with _load returning load_path (should probably return the file it self but getting a path from _io.Readerbuffer seemed like some work). _save is just a pass since I'll never change the file. Maybe there should be some generic any type of file for these sort of use cases, but it's the first time in my 6 months with Kedro that I've encountered this issue so it feels like an edge case