Fixit 2: Meta’s next-generation auto-fixing linter

• Fixit is useless! Lengthy stay Fixit 2 – the most recent model of our open-source auto-fixing linter.
• Fixit 2 permits builders to effectively construct customized lint guidelines and carry out auto-fixes for his or her codebases.
• Fixit 2 is out there immediately on PyPI.

Python is likely one of the hottest languages in use at Meta. Meta’s manufacturing engineers (PEs) are specialised software program engineers (SWEs) who give attention to reliability, effectivity, and scalability. They work on numerous initiatives, together with debugging manufacturing companies, rewriting inefficient libraries, orchestrating challenge deployments at scale, or capability planning and scheduling. And Python is usually one of many first instruments that PEs attain for, because it affords speedy growth, straightforward to learn syntax, and an enormous array of open supply libraries.

Meta’s Python Language Basis workforce — a hybrid workforce of each PEs and conventional SWEs — helps personal and preserve the infrastructure and tooling for Python at Meta. The workforce helps engineers, knowledge scientists, researchers, and anybody else at Meta utilizing Python to get their work performed.

One of many methods we accomplish that is constructing instruments that allow Python builders to jot down higher, and extra dependable code extra effectively. This contains instruments like automatic formatting and import sorting that eradicate tedium, or linters that information engineers towards maintainable code with fewer bugs.

This yr, we’ve been constructing a brand new linter, Fixit 2, designed from the bottom as much as make builders extra environment friendly and succesful, each in open supply initiatives and the varied panorama of our inner monorepo. At Meta, we’re utilizing Fixit 2 with a number of early adopters, and plan to roll it out to the remainder of our monorepo quickly. However any developer can use it to carry out auto-fixing extra effectively and make quicker enhancements to their very own codebases.

Why a brand new linter? (why not X?)

There are a number of wonderful linters within the Python ecosystem, lots of which have a big neighborhood of third-party plugins offering a various array of lint guidelines. We have now used Flake8 internally at Meta since 2016, and it has been very profitable in serving to builders scale back bugs and preserve a clear codebase. The favored flake8-bugbear plugin was even created by Łukasz Langa (writer of Black, PSF developer-in-residence, and launch supervisor for Python 3.8 and three.9) whereas working at Meta (then Fb), as a house for extra opinionated lint guidelines that we may each use internally and share with the remainder of the Python developer neighborhood. 

We even have a lot of inner plugins constructed by numerous groups, and Flake8 permits them to jot down and allow customized lint guidelines instantly within the codebase with out getting sign-off from a central gatekeeper, and with out ready for a brand new deployment of Flake8 to roll out.

However whereas Flake8 has lengthy been a cornerstone of our linting resolution, it additionally has some tough edges. Writing new lint guidelines requires constructing complete plugins (every claiming a portion of the “namespace” for error codes) and encourages builders to construct sophisticated plugins protecting a number of lessons of errors. When these lint errors are discovered, Flake8 can solely level to a line and column quantity the place it occurred, however has no method of suggesting modifications to the developer taking a look at a listing of lint outcomes, leaving them in a state of trial and error to search out modifications that make the linter blissful. Additionally, Flake8 makes use of the stdlib ast module, making it unable to parse future syntax options and forcing builders to attend for instruments to improve earlier than they will use the shiny new hotness.

There are options to Flake8 after all, however lots of them endure from a number of drawbacks: 

• A scarcity of help for “native” in-repo plugins or customized lint guidelines. 
• Restricted or no help for hierarchical configuration for various initiatives inside a monorepo.
• No possibility for auto-fixes when errors are discovered.
• Sluggish efficiency on massive codebases.

Whereas a few of these options aren’t vital, an important for developer effectivity is providing auto-fixes – computerized instructed modifications that will fulfill the lint rule. This takes the guesswork out of utilizing a linter, and permits customers to shortly assessment and settle for these modifications when doable, eliminating the necessity to re-run the linter till the code is lastly clear. Combining these auto-fixes with in-repo, customized lint guidelines supplies a stage of tailor-made code high quality enhancements that’s onerous to beat.

Sadly, even Fixit, the auto-fixing linter that we constructed for Instagram and open sourced, didn’t help native lint guidelines or hierarchical configuration – core necessities for our monorepo that’s dwelling to 1000’s of initiatives, lots of that are themselves open supply initiatives with their very own distinct wants for linting and CI. We obtained many requests from builders to help Fixit in our monorepo, however there have been sufficient hurdles that we have been solely capable of help a small set of safety lint guidelines, decreasing the direct advantages to our Python codebase. 

Meet Fixit 2

After discussions with different groups, particularly within the quickly rising AI/ML house, we thought of our choices and determined upon a partial rewrite of Fixit. We deliberately designed the brand new model with an open source-first mindset, whereas incorporating the wants and necessities of our personal monorepos and open supply initiatives from day one.

The framework and linting engine could be rebuilt from the bottom up whereas leaving the core design of lint guidelines largely untouched. The brand new system supplies a hierarchical configuration based mostly on the TOML format; help for native, in-repo lint guidelines just like Flake8; and a a lot improved CLI and API for integration with different instruments and automation. 

Fixit itself builds on high of one other Instagram open supply challenge, LibCST, a concrete syntax tree for Python with a tree and node API following the patterns of the ast module in the usual library. The “concrete” a part of CST signifies that LibCST contains each a part of the supply file within the ensuing tree after parsing, together with whitespace, feedback, and formatting parts which can be ignored by the ast module. That is what permits Fixit (and different instruments we constructed, like µsort) to soundly modify supply information, with out utilizing common expressions or the chance of manufacturing damaged syntax, and supplies the inspiration for Fixit to supply auto-fixes instructed by the lint guidelines themselves.

Writing a brand new lint rule will be performed with lower than a dozen traces of code, and check circumstances are outlined inline. You may even place it proper subsequent to the code that it will likely be linting:

# teambread/guidelines/hollywood.py
import fixit
import libcst
class HollywoodName(fixit.LintRule):
    VALID = [...] # no lint errors right here
    INVALID = [...] # dangerous code samples right here
    def visit_SimpleString(self, node: libcst.SimpleString):
        if node.worth in ('"Paul"', "'Paul'"):
            self.report(node, "It is underbaked!")

Suggesting auto-fixes for the person is as straightforward as together with a brand new CST node when reporting an error:

def visit_SimpleString(self, node: libcst.SimpleString):
    if node.worth in ('"Paul"', "'Paul'"):
        new_node = libcst.SimpleString('"Mary"')
        self.report(node, new_node)

Enabling this new rule inside the challenge’s codebase will be performed with a easy config change:

# teambread/sourdough/fixit.toml
[tool.fixit]
allow = [".rules.hollywood"]

Now we are able to run our linter towards our challenge:

# teambread/sourdough/baker.py
identify = "Paul"
print(f"howdy identify!")

$ fixit lint --diff sourdough/baker.py
sourdough/baker.py@7:11 HollywoodName: It is underbaked! (has autofix)
--- a/baker.py
+++ b/baker.py
@@ -6,3 +6,3 @@
def predominant():
-    identify = "Paul"
+    identify = "Mary"
    print(f"howdy identify")
🛠️  1 file checked, 1 file with errors, 1 auto-fix out there 🛠️
[1]

The `lint` command solely exhibits errors and instructed modifications. If we use the `repair` command, we are able to apply these instructed modifications again to the codebase:

$ fixit repair --automatic sourdough/baker.py
sourdough/baker.py@7:11 HollywoodName: It is underbaked! (has autofix)
🛠️  1 file checked, 1 file with errors, 1 auto-fix out there, 1 repair utilized 🛠️

Now that our auto-fixes have been utilized, we are able to affirm that the challenge is now clear and lint-free:

$ fixit lint sourdough/baker.py
🧼 1 file clear 🧼

When operating Fixit 2 with auto-fixing lint guidelines, any code that triggers the lint rule is a chance to get an computerized substitute, bettering the codebase with much less effort from the developer. Utilized extra broadly, Fixit 2 may even be used as a software to enact sweeping codemods towards a big codebase, whereas leaving a lint rule in place to deal with any matching code sooner or later.

Strive Fixit 2 

Fixit 2 is on the market immediately on PyPI. You may set up and check Fixit 2 with pip set up fixit. 

We have now a roadmap with plans for future enhancements and options, and a wealthy set of documentation and user guides that will help you get began with Fixit 2 in your personal initiatives or repositories. We hope it proves helpful in your initiatives, and we stay up for hearing your feedback!