amos
02/24/2022, 6:12 PMdatajoely
02/24/2022, 6:13 PMbeats-like-a-helix
02/25/2022, 3:01 PMbeats-like-a-helix
02/25/2022, 5:48 PMhttps://www.youtube.com/watch?v=mPPLk4IKu_sβΎ
datajoely
02/25/2022, 5:50 PMdatajoely
02/25/2022, 5:50 PMbeats-like-a-helix
02/25/2022, 6:31 PMmain()
function that looped over all the files in the directory. Many of my Astro projects are of a similar format to this, so been wondering what the best design choices are for a Kedro implementationdatajoely
02/25/2022, 6:35 PMbeats-like-a-helix
02/25/2022, 6:39 PMdatajoely
02/25/2022, 6:42 PMbeats-like-a-helix
02/25/2022, 6:47 PMwulfcrona
02/28/2022, 12:43 PMlbonini
02/28/2022, 2:19 PMdatajoely
02/28/2022, 2:23 PMafter_pipeline_created
hook that replaces the dataset with a MemoryDataSet dynamically based on the parameter or env variablelbonini
02/28/2022, 2:26 PMbeats-like-a-helix
03/04/2022, 7:17 PMbeats-like-a-helix
03/04/2022, 8:32 PMpython
plots_dict = dict()
for colour in ["blue", "green", "red"]:
plots_dict[f"{colour}.pdf"] = plt.figure()
plt.plot([1, 2, 3], [4, 5, 6], color=colour)
plt.close("all")
dict_plot_writer = MatplotlibWriter(
filepath="matplotlib_dict",
save_args={
# "format": "pdf",
"dpi": 300,
"bbox_inches": "tight",
},
)
dict_plot_writer.save(plots_dict)
pypeaday
03/04/2022, 9:37 PMdesrame
03/04/2022, 10:14 PMdesrame
03/04/2022, 10:15 PMbeats-like-a-helix
03/04/2022, 10:17 PMbeats-like-a-helix
03/04/2022, 10:31 PMyml
power_spectrum_figures:
type: PartitionedDataSet
path: data/07_model_output/figures
dataset:
type: matplotlib.MatplotlibWriter
save_args:
format: pdf
dpi: 300
bbox_inches: tight
filename_suffix: ".pdf"
beats-like-a-helix
03/04/2022, 10:37 PMsh
.
βββ README.md
βββ conf
βββ data
βββ docs
βββ info.log
βββ logs
βββ notebooks
βββ pyproject.toml
βββ setup.cfg
βββ src
beats-like-a-helix
03/04/2022, 10:42 PMdesrame
03/04/2022, 10:44 PMdesrame
03/04/2022, 10:45 PMdesrame
03/05/2022, 3:25 AMdesrame
03/05/2022, 3:26 AMdesrame
03/05/2022, 3:28 AM# catalog.yml definition
lstm_base:
type: pandas.SQLTableDataSet
table_name: sometable
credentials: sql1
# credentials.yml in local
sql1:
con: mssql+pyodbc:///?odbc_connect=DRIVER={ODBC+Driver+17+for+SQL+Server};SERVER=someserver;DATABASE=somedatabase;UID=someuser;PWD=somepwd
# .py version working in .py script
conn = \
'DRIVER={ODBC Driver 17 for SQL Server};SERVER=someserver;DATABASE=somedatabase;UID=someuser;PWD=somepassword'
quoted = quote_plus(conn)
new_con = 'mssql+pyodbc:///?odbc_connect={}'.format(quoted)
engine = create_engine(new_con, fast_executemany=True, connect_args={'timeout': 100})
desrame
03/05/2022, 3:28 AMdesrame
03/05/2022, 3:28 AM