Arnaldo
12/01/2021, 4:56 PMdatajoely
12/13/2021, 10:19 AMdatajoely
12/13/2021, 10:25 AMpandas.ExcelDataSet
can now save files with multiple sheets
📜 pandas.SQLQuerydataSet
can accept the query through a file (via a new filepath
argument)
🗑️ PartitionedDataSet
and MatplotlibWriter
can now delete existing partition/plots on save()
🥒 Added support for arbitrary backends for pickle.PickleDataSet
🗄️ Added 4 new datasets, including:
* pandas.GenericDataSet
to read/write any format supported by pandas
* spark.DeltaTableDataSet
for handling DeltaLake tables
➕ Added support for sum()
syntax for connecting Pipeline
objects
⚡ Made some performance optimisations when loading up the catalog
🐛 Fixed some bugs with kedro pipeline package
and kedro pipeline pull
🐞 Removed unintentional dependency on git
See the for more info.
💻 Installation
Kedro 0.17.6 is a non-breaking release in the 0.17.x series, upgrade now with:
pip install kedro==0.17.6
conda install -c conda-forge kedro==0.17.6
datajoely
12/13/2021, 5:46 PMCVE-2021-44228
log4j RCE - Kedro supply chain is not affected
- Kedro supports PySpark pyspark>=2.2, <4.0
as an optional dependency
- All versions of PySpark (and Hive use) use log4j version 1.2.17
out of the box
- The issue affects versions log4j >2.10
and is fixed in 2.15.0
- Unless your cluster has manually updated your version of logger you should not be affected (ironically older version helps here)Yetunde
01/20/2022, 11:29 AMdatajoely
01/20/2022, 11:29 AMdatajoely
01/20/2022, 12:07 PMdatajoely
01/21/2022, 12:47 PMdatajoely
02/14/2022, 10:31 AMkedro-dvc
something we've been keen to see for aaaaages, if you want to get involved there are lots of conversations happening on this issue on our side:
https://github.com/kedro-org/kedro/discussions/837#discussioncomment-2154419
and on their repo:
https://github.com/FactFiber/kedro-dvcSajidAlamQB
02/22/2022, 5:50 PMpandas<1.4
to maintain compatibility with xlrd~=1.0
.
- Bumped the Pillow
minimum version requirement to 9.0 (Python 3.7+ only) following CVE-2022-22817
.
- Fixed PickleDataSet
to be copyable and hence work with the parallel runner.
- Upgraded pip-tools
, which is used by kedro build-reqs
, to 6.5 (Python 3.7+ only). This pip-tools
version is compatible with pip>=21.2
, including the most recent releases of pip
. Python 3.6 users should continue to use pip-tools
6.4 and pip<22
.
- Added astro-iris
as alias for astro-airlow-iris
, so that old tutorials can still be followed.
- Added details about Kedro's Technical Steering Committee and governance model.
Major features:
- pipeline
now accepts tags
and a collection of `Node`s and/or `Pipeline`s rather than just a single Pipeline
object. pipeline
should be used in preference to Pipeline
when creating a Kedro pipeline.
- pandas.SQLTableDataSet
and pandas.SQLQueryDataSet
now only open one connection per database, at instantiation time (therefore at catalog creation time), rather than one per load/save operation.
- Added new command group, micropkg
, to replace kedro pipeline pull
and kedro pipeline package
with kedro micropkg pull
and kedro micropkg package
for Kedro 0.18.0.kedro micropkg package
saves packages to project/dist
while kedro pipeline package
saves packages to project/src/dist
.
Kedro 0.17.7 is a non-breaking release in the 0.17.x series, upgrade now with:
pip install kedro==0.17.7
conda install -c conda-forge kedro==0.17.7
Arnaldo
02/22/2022, 7:15 PMelvis
02/22/2022, 7:31 PMNero_Okwa
04/01/2022, 3:36 PMTL;DR
In this release we reduce the complexity of the project template to get us closer to a stable release of the framework. We've introduced the full micro-packaging workflow 📦, which allows you to import packages, utility functions and existing pipelines into your Kedro project. Integration with IPython and Jupyter has been streamlined in preparation for enhancements to Kedro's interactive workflow. Additionally, the release comes with long-awaited Python 3.9 and 3.10 support 🐍. Please make sure to trial our experiment tracking workflow in Kedro.
⚠️ There are breaking changes in this release, you can read the full release notes: https://github.com/kedro-org/kedro/releases
What can you expect in this release?
- The micro-packaging workflow 📦,which allows you to package and export Kedro pipelines, Python packages, utility functions, and modules in your code base, to your choice of Python package repository.
- The only recommended way to work with Kedro in Jupyter or IPython is now the Kedro IPython extension.
- Reduced boilerplate and complexity of the project template. We have removed cli.py
from the Kedro project template.
- A packaged Kedro project will respect the same hierarchy when executed with python -m my_package
.
- Added Python 3.9 and 3.10 support 🐍 .Dropped support for Python 3.6.
- Refactored the load and save operations for pandas
datasets in order to leverage pandas
own API and delegate fsspec
operations to them. This reduces the need to have our own fsspec
wrappers, like in previous versions.
How do I get access to the latest release?
Get the latest version of Kedro with pip install kedro == 0.18.0
and await being able to install Kedro from Conda with conda install -c conda-forge kedro==0.18.0
.Nero_Okwa
04/01/2022, 3:38 PMtynan
04/05/2022, 5:36 PMexpandAllPipelines
feature flag that expands all modular pipelines by default. Turn this on by clicking the settings icon ⚙️
How do I get access to the latest release?
Get the latest version of Kedro-Viz with pip install kedro-viz==4.4.0
(Python) or with npm install @quantumblack/kedro-viz@latest
(React)
What's next?
Python 3.9 and 3.10 support will come in a following patch release. We'll also update our demo around that time too.
Until next time,
The Kedro Team 💜
Footnotes:
- #1: https://github.com/kedro-org/kedro-viz/releases/tag/v4.4.0
- #2: https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html
Merel
04/14/2022, 10:30 AMArnaldo
04/14/2022, 12:44 PMNero_Okwa
05/05/2022, 9:44 AMexperiment tracking
. Kindly respond with a 👍 if you have used this feature, or know someone who has. Thank you.Merel
05/10/2022, 11:20 AMCONFIG_LOADER_CLASS
validation so that `TemplatedConfigLoader `can be specified in settings.py. Any CONFIG_LOADER_CLASS
must be a subclass of AbstractConfigLoader
.
- Fixed Jinja2
syntax loading with TemplatedConfigLoader
using globals.yml.
- Removed global _active_session
, _activate_session
and _deactivate_session
. Plugins that need to access objects such as the config loader should now do so through context in the new after_context_created
hook.
- Made hook_manager
argument optional for runner.run.
- kedro docs now opens an online version of the Kedro documentation instead of a locally built version.
🚀 Major features:
- Added a new hook after_context_created
that passes the KedroContext
instance as context
.
- Added a new CLI hook after_command_run
.
- The Kedro package no longer contains a built version of the Kedro documentation significantly reducing the package size 🐳
You can find the full list of changes in the release notes: https://github.com/kedro-org/kedro/blob/main/RELEASE.md
Kedro 0.18.1 is a non-breaking release in the 0.18.x series, upgrade now with:
pip install kedro==0.18.1
conda install -c conda-forge kedro==0.18.1
Yetunde
05/24/2022, 2:23 PMWolVez
05/24/2022, 2:35 PMtynan
06/17/2022, 9:27 AMmatplotlib
images in the Metadata panel and Modal 📊
- Support for all plotly
chart types including Sankey diagrams 📈
- Python 3.9 and 3.10 support
- A whole host of other UX/UI improvements and bug fixes, too 🐛
How do I get access to the latest release?
- Get the latest version of Kedro-Viz with pip install kedro-viz==4.7.0
(Python) or with npm install @quantumblack/kedro-viz@latest
(React)
What's next?
- An easier way to run Kedro-Viz in platforms like Databricks without a CLI and further honing of the application experience.
- Improvements on Experiment Tracking (#3), like being able to track plots and datasets with experiments
Until next time,
The Kedro Team 💜
Footnotes:
- #1: https://github.com/kedro-org/kedro-viz/releases/tag/v4.7.0
- #2: https://demo.kedro.org/
- #3: https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html
Ahdra
07/12/2022, 9:23 AMconf/base/logging.yml
is now optional. See our documentation for details.
- Introduced a kedro.starters
entry point. This enables plugins to create custom starter aliases used by kedro starter list
and kedro new
.
- Reduced the kedro new
prompts to just one question asking for the project name.
🐛 Bug fixes:
- Bumped pyyaml
upper bound to make Kedro compatible with the pyodide
stack.
- Kedro's framework-side logging configuration no longer performs file-based logging. Hence superfluous `info.log`/`errors.log `files are no longer created in your project root, and running Kedro on read-only file systems such as Databricks Repos is now possible.
- Required cookiecutter>=2.1.1
to address a known command injection vulnerability.
- The session store no longer fails if a username cannot be found with getpass.getuser
.
- Added generic typing for AbstractDataSet
and AbstractVersionedDataSet
as well as typing to all datasets.
To make the most of our new rich logging you can either delete conf/base/logging.yml
if it has not been customised, or add rich.logging.RichHandler
to the logging handlers. See the logging documentation for more.
You can find the full list of changes in the release notes: https://github.com/kedro-org/kedro/blob/main/RELEASE.md
Kedro 0.18.2 is a non-breaking release in the 0.18.x series, upgrade with:
pip install kedro==0.18.2
conda install -c conda-forge kedro==0.18.2
tynan
08/05/2022, 1:53 PMpip install kedro-viz==5.0.0
(Python) or with npm install @quantumblack/kedro-viz@latest
(React)
What's next?
- Improvements on Experiment Tracking (#3), like being able to track and plot metrics over time.
- And follow our roadmap (#4) for more updates.
Until next time,
The Kedro Team 💜
Footnotes:
- #1: https://github.com/kedro-org/kedro-viz/releases/tag/v5.0.0
- #2: https://demo.kedro.org/
- #3: https://kedro.readthedocs.io/en/stable/tutorial/set_up_experiment_tracking.html
- #4: https://github.com/orgs/kedro-org/projects/3/views/1
user
08/15/2022, 12:14 AMuser
09/13/2022, 4:04 AMJose Alejandro M
09/13/2022, 8:47 PMuser
09/19/2022, 3:29 AMnoklam
09/20/2022, 3:37 PM%reload_kedro
now accepts keywords arguments. There are couple of fixes to make Rich’s log less verbose and work better with Databricks.
Kedro 0.18.3 is a non-breaking release in the 0.18.x series, which means you can upgrade with:
pip install kedro==0.18.3
or conda install -c conda-forge kedro==0.18.3
🚀 Major Features/Changes
- [NEW] - autodiscovery of project pipelines. See our documentation for more details.
- The Kedro IPython extension should now be loaded with %load_ext kedro.ipython.
- The line magic %reload_kedro
now accepts keywords arguments, e.g. %reload_kedro --env=prod.
- Improved resume pipeline suggestion for SequentialRunner
, it will backtrack the closest persisted inputs to resume.
🪲 Bug fixes and other changes
- Changed default False
value for rich logging show_locals
, to make sure credentials and other sensitive data isn’t shown in logs.
- Rich traceback handling is disabled on Databricks so that exceptions now halt execution as expected.
- When using kedro run -n [some_node]
, if some_node
is missing a namespace the resulting error message will suggest the correct node name.
- The Kedro IPython extension now surfaces errors when it cannot load a Kedro project.
- And more …
You can find the full list of changes in the release notes:
https://github.com/kedro-org/kedro/releases/tag/0.18.3