ATProto Browser

Record data

{
  "uri": "at://did:plc:7oyzfpde4xg23u447zkp3b2i/com.whtwnd.blog.entry/3lnjm3t5ijb27",
  "cid": "bafyreiefablswezuxm33igfjgp4xorkheek4fwmikmrbxhxbh6lcnnlz5y",
  "value": {
    "$type": "com.whtwnd.blog.entry",
    "theme": "github-light",
    "title": "On the origin of Resyntax",
    "content": "I've been working on [Resyntax](https://docs.racket-lang.org/resyntax/), my automated refactoring tool for [Racket](https://racket-lang.org/), for over four years now.\n\nIt all began one evening in January 2021 when, after years of pondering the question of automated refactoring tooling for Racket, I asked myself a very useful question:\n\n> \"Wait, how the hell does DrRacket's [Macro Stepper](https://docs.racket-lang.org/macro-debugger/) actually work?\"\n\nI've revisited this question several times over the years. Each time, I've learned something frightening and powerful.\n\n## First encounters with eldritch powers\n\nThe first time I asked that question, I decided to dig around in DrRacket's codebase in search of answers. I was initially confused because if you crawl through the [enormous reference documentation](https://docs.racket-lang.org/reference/Macros.html) on Racket's macro system and [syntax model](https://docs.racket-lang.org/reference/syntax-model.html), it isn't immediately obvious how one could inspect the complex intermediate states of the macro expander. You could manually step through execution repeatedly with [`expand-once`](https://docs.racket-lang.org/reference/Expanding_Top-Level_Forms.html#%28def._%28%28quote._~23~25kernel%29._expand-once%29%29), but this gives you pretty limited information - certainly a lot less than the Macro Stepper reveals. So if DrRacket, adhering to the Racket philosophy of not privileging external tools with [extralinguistic mechanisms](https://felleisen.org/matthias/manifesto/sec_intern.html), is able to keep track of all this state and expose it to users... then there _must_ be some API somewhere that the powers that be Don't Want You To Know About (because it's unstable and a maintenance headache to commit to).\n\nAnd I found it. It's called `current-expand-observe` and it lives in the `#%expobs` kernel module. Here's [the macro stepper grabbing ahold of it through dynamic shenanigans](https://github.com/racket/macro-debugger/blob/047c36a4d6a0b15a31d0fa7509999b9bf2c5f295/macro-debugger/macro-debugger/model/trace-raw.rkt#L9).\n\n```\n(define current-expand-observe\n  (dynamic-require ''#%expobs 'current-expand-observe))\n```\n\nThis parameter (in the Racket sense of a [parameter](https://docs.racket-lang.org/guide/parameterize.html), not a normal function parameter) allows one to set the current _macro expansion event observation hook_, which is a function that receives _expansion events_ from the Racket macro expander every time it does something. Each event comes with a payload; usually some [syntax objects](https://docs.racket-lang.org/guide/stx-obj.html) and identifiers relevant to the event in question. The specific events and event payloads aren't documented and are far from stable, as they change whenever the expander's implementation details warrant. You can find all of the places where Racket emits these events by [searching the Racket implementation for `log-expand`](https://github.com/search?q=repo%3Aracket%2Fracket%20log-expand&type=code).\n\n## Background: two core problems for refactoring macros\n\nWith this occult tool in hand, I realized I could fairly easily put together a refactoring tool that expands code and keeps track of all of the pieces of code that the expander actually _visited_. This is important, because macros and [quotation](https://docs.racket-lang.org/guide/quote.html) mean that it's pretty common to see code that _looks_ like real code but which is actually data. Consider this code:\n\n```\n(define (build-square-expression x)\n  `(* ,x ,x)\n```\n\nA naive refactoring tool - even one based on AST transformations - might lodge a complaint here, asking the user why they didn't rewrite `(* ,x ,x)` to `(sqr ,x)`. But in our case, the code in question _isn't code_. It's quoted data. (More precisely, it's [quasiquoted](https://docs.racket-lang.org/guide/qq.html) data.) Macros introduce similar problems via [syntax templates](https://docs.racket-lang.org/reference/stx-patterns.html#%28form._%28%28lib._racket%2Fprivate%2Fstxcase-scheme..rkt%29._syntax%29%29), and ensuring a refactoring tool can handle macros is very important because  Racketeers absolutely love their macros.\n\nWe can't even try to work around the problem by writing rules that try to detect when the AST node they're analyzing is underneath a quote. While direct use of quotes can be caught in this manner, as Racketeers we must reckon with the unfortunate truth that the use of quasiquotation could be _hidden by a macro_. For example:\n\n```\n(define-syntax-rule (my-quote expr)\n  (quasiquote expr))\n\n(define (build-square-expression x)\n  (my-quote (* ,x ,x)))\n```\n\nIn order to tell that `(* ,x ,x)` is quoted data and not actual code, we have to expand the `my-quote` macro. Without expansion, we cannot be certain that the code we're looking at is what it appears to be.\n\nWe could use the expander directly and try to analyze the [fully expanded code](https://docs.racket-lang.org/reference/syntax-model.html#%28part._fully-expanded%29), but that falls prey to a different problem. A refactoring tool wants to refactor code _written by humans_, not code _generated by macros_. Refactoring is meant to make code more readable to humans. If humans aren't reading the code in the first place, as is the case for macro-generated code, there's hardly any point in refactoring it. You don't see the macro-generated code, so why would you care how pretty it looks?\n\nFurthermore, some code might only be needed during expansion. This code effectively vanishes after expansion and won't exist in the fully expanded code. This is usually because the code was used at compile-time, not runtime. If we only examine the fully expanded code, we won't be able to refactor such \"disappeared uses\".\n\nSo that's our two big problems: telling the difference between code and data, and telling the difference between code written by humans and code generated by macros.\n\n## The genesis of Resyntax\n\nWith `current-expand-observe`, we now have a solution for both of the above problems. We can tell which code is _code_ and which code is _data_ by tracking which forms the expander actually expands. Data doesn't get expanded, because data doesn't _do_ anything on its own. We can also keep track of code at _all_ steps of the expansion, not just the final expansion result: this ensures that even though we're expanding the code, we still have visibility into code that might vanish during expansion. To tell what code is spit out by macros, we can use [`syntax-original?`](https://docs.racket-lang.org/reference/stxops.html#%28def._%28%28quote._~23~25kernel%29._syntax-original~3f%29%29) on the visited syntax objects to determine if the expanded form was produced by a macro or not.\n\nDark magic in hand, I put `current-expand-observe` to use and built the [initial Resyntax prototype](https://github.com/jackfirth/resyntax/commit/75423aae822dd69910978ccdbcc4cb448371d3cb#diff-9d629e9198e03ca0a3046f8f44e650d4f7e3b650ca2807ec982977b3c1c34797) on January 8th, 2021. I only had a tiny handful of basic refactoring rules, really just enough to verify that the system worked at all. Which it didn't. I'd forgotten a `require` import in the example code and had to fix that in the [next commit](https://github.com/jackfirth/resyntax/commit/5b8e98bfd548c22416f04d80ed84249a84ab23ee). But _then_ it worked, and there was much rejoicing.\n\n## Where things went\n\nAfter that fateful day, the rest was history. Resyntax has grown a lot of new functionality since then. [Over a hundred built-in refactoring rules](https://github.com/jackfirth/resyntax/tree/9d5d1815150f457e55909f96f33483eb51bf19ac/default-recommendations), an [entire DSL for testing them](https://github.com/jackfirth/resyntax/blob/9d5d1815150f457e55909f96f33483eb51bf19ac/default-recommendations/hash-shortcuts-test.rkt), an [extensible API for creating rules](https://docs.racket-lang.org/resyntax/Refactoring_Rules_and_Suites.html), a [command-line interface](https://docs.racket-lang.org/resyntax/cli.html) for analyzing and refactoring code, an [Autofixer](https://github.com/jackfirth/resyntax-autofixer) that now generates [weekly pull requests](https://github.com/racket/drracket/pull/735) to several Racket repositories, and more. But the core has remained unchanged: Resyntax expands your code using Racket's macro expander while listening to Racket for metadata in expansion events, and then Resyntax uses that information to apply its refactoring rules to your code in an intelligent and binding-aware manner.\n\nIt's amazing what Racket's macro expander can do.",
    "createdAt": "2025-04-24T03:21:48.141Z",
    "visibility": "public"
  }
}