Wit
05/10/2022, 8:01 PMWit
05/10/2022, 8:02 PMWit
05/10/2022, 8:03 PMWit
05/10/2022, 8:03 PMdatajoely
05/10/2022, 8:07 PMmarioFeynman
05/11/2022, 12:59 AMmarioFeynman
05/11/2022, 1:01 AMRjify
05/11/2022, 5:36 AMinigohrey
05/11/2022, 8:49 AMTsakagur
05/11/2022, 9:44 AMYetunde
05/11/2022, 10:25 AMkedro package
and publishing the package using the Databricks DBFS API
- Using Databricks Repos functionality and doing a pipeline run through a Databricks notebookxxavier
05/11/2022, 12:25 PMrun_histograms:
type: api.APIDataSet
url: https://xxx/
headers:
Authorization: Token <token>
Works without error (which is nice but token is somehow sensitive information).
I tried to fill the header using credentials but failed to do so.
credentials.yml
dqm_playground_token:
- Content-Type: application/json
- Authorization: Token <token>
catalog.yml
run_histograms:
type: api.APIDataSet
url: https://xxx/
# Test 1
headers: dqm_playground_token
# Test 2
headers:
- dqm_playground_token
# More tests
It seems to boil down to the fact that it reads Dict[str, Any] and not Union[Iterable[str], AuthBase]:
https://kedro.readthedocs.io/en/stable/_modules/kedro/extras/datasets/api/api_dataset.html#APIDataSet
I could probably modify the APIDataSet definition to solve it by having headers = auth but I guess there is a better way. 🙂 Sorry about the naive question.
Any help is appreciated.datajoely
05/11/2022, 12:53 PMdatajoely
05/11/2022, 12:54 PMdatajoely
05/11/2022, 12:55 PMmarioFeynman
05/11/2022, 1:30 PMdatajoely
05/11/2022, 1:31 PMxxavier
05/11/2022, 2:15 PMrun_histograms:
type: dqm_playground_ds.extras.datasets.tuned_API_dataset.TunedAPIDataSet
url: https://xxx/
credentials: dqm_playground_token
headers: credentials
credentials.yml
dqm_playground_token:
- Authorization: Token <token>
tuned_API_dataset.py (similar to kedro's api_dataset.py)
python3
auth = credentials or auth
# Added the following three lines :)
if headers == "credentials":
auth = None
headers = credentials[0]
Not great, not terrible. 🙂 Thanks again!marioFeynman
05/11/2022, 3:51 PMRjify
05/11/2022, 4:26 PMRjify
05/11/2022, 9:04 PMdatajoely
05/11/2022, 9:05 PMdatajoely
05/11/2022, 9:06 PMuser
05/12/2022, 2:33 AMmarioFeynman
05/13/2022, 1:18 AMantony.milne
05/13/2022, 7:45 AMKastakin
05/14/2022, 10:55 AMKastakin
05/14/2022, 10:56 AMnoklam
05/15/2022, 5:49 PMnoklam
05/15/2022, 5:49 PM