I think it is important to realize that going full in on agile for a system that aims to provide (or should aim to provide) High-Availability is a no-go. You need to strike a balance between the simulation, shadow-run and DTAP requirements posed by the HE setting and the desire for velocity that an agile CICD setup promises.
The important thing to realize is that being a blockchain project doesn't so much bring you into virgin territory on this as you might think. Modern non-blockchain high availability shops have been balancing the same scale for a long time and you guys should really find some friendly faces in shops like these to show you what they did to maintain HE while carefully moving to a slightly more agile approach.
Some things you might want to talk about with HE shop people:
- HE-DTAP
- Building representative high-volume test sets
- Building simulation/event-generator setups with model driven feedback loops
- Setting up an event-fork based simulation infrastructure for partial parallelization of the A and P.
- "A" centered CICD possibilities and limitations.
Basically, stop thinking you need to do things differently because you are a blockchain shop and realize you need to do things differently because you should be a HE shop. Talk to non-blockchain HE shops that have managed to combine aspects of CICD with HE-DTAP and see what you can learn from them.
My own knowledge on this is mostly centered around #3 and I'm most definitely no expert on the other four, so as much as I would like to help out on the bigger picture, my help won't be of much use until you guys get the bigger picture sorted. For that, some face to face time with a modern HE team I feel has the potential to really make a huge difference.
I see Steemit Inc is based in NY. I'm pretty sure there will be quite a few modern financial and tech sector HE shops there to provide you guys with some ideas on how to successfully marry Agile CICD with HE-DTAP. Try to find one or two willing to give you guys some insights into the way they try to strike this balance for their own shop.