Another issue is launching behind feature flags often skips integration/e2e testing, which couldn't possibly test with every combination of every flag. So your release which is "safer" behind a flag is actually untested until it's launched.
paulryanrogers 51 minutes ago [-]
Worked at a place with a literal settings table of 5K (non-default) records, it was many hundreds of possible settings.
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
MaulingMonkey 2 hours ago [-]
Feature flags are great. Working on a crash / memory corruption in an optional subsystem? Just disable the subsystem to unblock coworkers on the same branch while you track down the cause.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
joezydeco 2 hours ago [-]
"At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second"
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
classictraffic 3 hours ago [-]
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
llIIllIIllIIl 3 hours ago [-]
I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software.
So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
lgrthmsprs 2 hours ago [-]
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
sgarland 54 minutes ago [-]
As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week.
Just use canaries, I’m begging you.
willsmith72 37 seconds ago [-]
good luck rolling back when 20 teams have been waiting for your pipeline to be unblocked for 2 weeks
paulryanrogers 47 minutes ago [-]
Canaries meaning deploying the new stuff only to certain segments first?
I imagine that's got its own risks and challenges if there is a lot of bottlenecks or dependencies, like multi-tenant data store(s).
Rendered at 05:03:41 GMT+0000 (Coordinated Universal Time) with Vercel.
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
1 feature flag = 2 behaviors 2 feature flags = 4 behaviors
So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
Just use canaries, I’m begging you.
I imagine that's got its own risks and challenges if there is a lot of bottlenecks or dependencies, like multi-tenant data store(s).