Thanks for putting together a plan, it's a great start. Answering to your points:
> I’d like to move bench_runner into the Python org on GitHub
Yes please, I know there are many forks/versions around, it would be great to have one that is well maintained.
> I think it would be valuable to have more recognizable macrobenchmarks
I totally agree with it. Maybe we can start looking at pyston/python-macrobenchmarks
IIRC, folks ran those as well when evaluating CPython performance
> there are benchmarks folks intentionally ignore because they don’t think they’re useful or good benchmarks
yes, we should remove them. If we don't want to remove them, we should group them together under the noisy group :slight_smile:
Internally we skip concurrent_imap, unpack_sequence.
> Nightly runs
These can be done if we have a reliable infrastructure underneath. An yes, speed.python.org should be replaced with something more modern and useful
> A perf bot
I support the idea to have automation to request performance on demand. Also it would be good to have the bot to run it with pystats, so it can collect stats about a specific change.
Making performance work easier in CPython
After chatting with folks at PyCon US, EuroPython and the sprint last week, I wanted to start a broader discussion about our benchmarking infrastructure and tooling. There are a few things I’d like to work on and would love input from folks.
Rehoming bench_runner into the Python org
I’d like to move bench_runner into the Python org on GitHub.
For those unfamiliar, bench_runner is what we use to run pyperformance through GitHub Actions on self-hosted runners. I’ve been using it for doesjitgobrrr.com for the better part of a year, and I know Meta and Arm have it configured as well.
This tool currently lives in the Faster CPython org, but I think it falls pretty squarely into Python org territory. I’d like to take on more of the maintenance and cleanup, including making the setup documentation easier to follow. I’ve documented my setup on my personal website to some degree, but it’d be good to bring that into the Devguide too.
I chatted with @mdroettboom about the move earlier this year, and he was supportive.
Our benchmark suite and workflow
This is a bigger and, I expect, more contentious topic, but there are a couple of things I’ve been thinking about:
- Our benchmark mix: Small, focused benchmarks are useful, but I think we’d benefit from revisiting the balance of those and larger workloads. I’m also interested in how families of related benchmarks contribute to the overall geometric mean. For example, are we giving the various async-tree benchmarks the right collective weight? I also know anecdotally that there are benchmarks folks intentionally ignore because they don’t think they’re useful or good benchmarks. It’d be good to hear which ones and why.
- Workloads: I think it would be valuable to have more recognizable macrobenchmarks: a modern web application, a data analysis workflow, an ML workload, etc. Having some agreed-upon workloads that we benchmark regularly across machines could help make performance improvements more tangible for people outside the project. It’s easier to connect a result to your own work when you recognize the application or workflow being measured. Richards, for example, probably doesn’t mean much to most end users.
- Nightly runs: The nightly runs for doesjitgobrrr.com have been really useful for catching regressions and seeing whether optimizations help (see @kj0's blog about this too). I think it could be valuable to build on this and make day-over-day performance tracking easier to follow for CPython more broadly (doespythongobrrr?). Maybe that eventually includes adapting the site into something that could replace speed.python.org.
A perf bot
There are a lot of times in PRs where we ask whether a change is worth it or what the performance delta actually is. Getting that answer can be slow and requires access to a benchmarking setup, or finding someone who has it.
I’d love something like a !perf command or a label on a PR that core team members could use to trigger benchmarks and get results compared to main. We’d probably need a subset of benchmarks or a more targeted run to make the turnaround useful during review. Details TBD, but something faster than two hours would be awesome.
So, a few questions to kick this off:
- Are you supportive of some or all of these ideas?
- For folks working on performance, what would help you get useful data faster?
- Which workloads should we be paying more attention to?
17 likes · 4 comments
Mentions
Thanks for putting together a plan, it's a great start. Answering to your points: > I’d like to move bench_runner into the Python org on GitHub Yes please, I know there are many forks/versions around, it would be great to have one that is well maintained. > I think it would be valuable to have more recognizable macrobenchmarks I totally agree with it. Maybe we can start looking at pyston/python-macrobenchmarks IIRC, folks ran those as well when evaluating CPython performance > there are benchmarks folks intentionally ignore because they don’t think they’re useful or good benchmarks yes, we should remove them. If we don't want to remove them, we should group them together under the noisy group :slight_smile: Internally we skip concurrent_imap, unpack_sequence. > Nightly runs These can be done if we have a reliable infrastructure underneath. An yes, speed.python.org should be replaced with something more modern and useful > A perf bot I support the idea to have automation to request performance on demand. Also it would be good to have the bot to run it with pystats, so it can collect stats about a specific change.
[quote="Savannah Ostrowski, post:1, topic:109711, username:savannahostrowski"] I’d like to move bench_runner into the Python org on GitHub. [/quote] Yes, let's do it.
Making performance work easier in CPython After chatting with folks at PyCon US, EuroPython and the sprint last week, I wanted to start a broader discussion about our benchmarking infrastructure and tooling. There are a few things I’d like to work on and would love input from folks. Rehoming bench_runner into the Python org I’d like to move bench_runner into the Python org on GitHub. For those unfamiliar, bench_runner is what we use to run pyperformance through GitHub Actions on self-hosted runners. I’ve been using it for doesjitgobrrr.com for the better part of a year, and I know Meta and Arm have it configured as well. This tool currently lives in the Faster CPython org, but I think it falls pretty squarely into Python org territory. I’d like to take on more of the maintenance and cleanup, including making the setup documentation easier to follow. I’ve documented my setup on my personal website to some degree, but it’d be good to bring that into the Devguide too. I chatted with @mdroettboom about the move earlier this year, and he was supportive. Our benchmark suite and workflow This is a bigger and, I expect, more contentious topic, but there are a couple of things I’ve been thinking about: - Our benchmark mix: Small, focused benchmarks are useful, but I think we’d benefit from revisiting the balance of those and larger workloads. I’m also interested in how families of related benchmarks contribute to the overall geometric mean. For example, are we giving the various async-tree benchmarks the right collective weight? I also know anecdotally that there are benchmarks folks intentionally ignore because they don’t think they’re useful or good benchmarks. It’d be good to hear which ones and why. - Workloads: I think it would be valuable to have more recognizable macrobenchmarks: a modern web application, a data analysis workflow, an ML workload, etc. Having some agreed-upon workloads that we benchmark regularly across machines could help make performance improvements more tangible for people outside the project. It’s easier to connect a result to your own work when you recognize the application or workflow being measured. Richards, for example, probably doesn’t mean much to most end users. - Nightly runs: The nightly runs for doesjitgobrrr.com have been really useful for catching regressions and seeing whether optimizations help (see @kj0's blog about this too). I think it could be valuable to build on this and make day-over-day performance tracking easier to follow for CPython more broadly (doespythongobrrr?). Maybe that eventually includes adapting the site into something that could replace speed.python.org. A perf bot There are a lot of times in PRs where we ask whether a change is worth it or what the performance delta actually is. Getting that answer can be slow and requires access to a benchmarking setup, or finding someone who has it. I’d love something like a !perf command or a label on a PR that core team members could use to trigger benchmarks and get results compared to main. We’d probably need a subset of benchmarks or a more targeted run to make the turnaround useful during review. Details TBD, but something faster than two hours would be awesome. So, a few questions to kick this off: - Are you supportive of some or all of these ideas? - For folks working on performance, what would help you get useful data faster? - Which workloads should we be paying more attention to?