On the same topic of environments, if you wanted to have a separate spark configuration for different nodes in a pipeline, is the correct approach to store those configurations in separate environments and execute the nodes in separate runs with -env flags? Or is there some other way that would allow changing the configuration, perhaps by reloading the context, within the same run?

So Kedro isn't designed to be execution environment aware like fully fledged orchestrators like Argo, Flyte etc

Generally agreed with Joel on this, but actually since kedro 0.18.1 introduced the

after_context_created

hook I wonder if there's a better way of doing it now... Here's a rough demo of how you could do it: https://gist.github.com/AntonyMilneQB/792a748b0d921e2f9f78cc7dd9c13c97. The advantage of this are: * no need for a custom

KedroContext

at all, since all the spark stuff is done in hooks * you can still use run environments as you currently do, no need to create a separate run environment for each spark config (although you still can do so if you like)

I also agree that much of this can/should be delegated to the orchestrator; however, for early developing stages or configuring spark params that aren't tied so tightly to cluster params (maybe num_shuffle_partitions or something?), I'm interested in trying out the hook-based approach you outline here. It seems potentially simpler to have multiple spark configs in the same environment and map to them via node names or tags. I haven't used hooks yet, so this gives me a good excuse 🙂

Let us know how this goes - as @antony.milne mentioned this is a brand new feature so any and all feedback is really appreciated

The more I think about it, the more I think that node tags is actually a very good way to do this. It's quite similar to this idea: https://discord.com/channels/778216384475693066/846330075535769601/935879323813036142.

I think for deployment I might keep most/all of the spark configuration outside kedro and instead include it in the spark-submit. During development, I don't mind yaml inside kedro. Some of the options in my current config are things like: spark.hadoop.fs.s3.canned.acl, spark.sql.adaptive.enabled, spark.sql.adaptive.coalescePartitions.enabled, spark.sql.adaptive.coalescePartitions.minPartitionSize, spark.sql.shuffle.partitions, spark.sql.adaptive.skewJoin.enabled

Thanks very much for the explanation! stupid question from someone who doesn't know much about spark, but when you say "include it in spark-submit", you mean you'd do something like this?

spark-submit --conf spark.hadoop.fs.s3.canned.acl=...  spark.sql.adaptive.enabled=...

Yes exactly. When we run a pipeline from airflow, we typically run spark tasks as EMR "steps" via a custom EMR operator which uses spark-submit. Also, there are no stupid spark questions, but I very well may give stupid answers!

Thank you, this is very helpful! The thing which I'm really interested in here is whether storing spark config in a spark.yml file is a natural thing to do, since I'm not sure where we originally got this idea from on Kedro. If you want to be able to specify a different spark configuration per run environment then using yaml is the typical Kedro way to do things, but I've always wondered whether anyone else would use yaml to specify spark config outside of Kedro. As far as I can tell spark config is just a set of key-value pairs (is that right?), so having a yaml file for it in one sense overkill because you don't need a general dictionary structure. On the other hand, just listing arguments like in

spark-submit

feels very awkward to me and I'm surprised there's no

--config-file

option already where you can input some file in a standardised format of key-value pairs. Like doesn't your

spark-submit

command get huge if you want to specify 100 options? Or does that never really happen?

Ah gotcha, yes the spark-submits can get large but we typically call them through a command-runner.jar that can accept a bunch of arguments to assemble the spark-submit call like in this pic.

Thank you very much for the explanation @bgereke, that's super helpful. Let me know how it goes and if you develop any more opinions on how we handle spark in kedro - I'm very interested in seeing if this is something we could improve.