"CL dorks" refer to Lisp as Lisp, because before you were born there were many Lisps: maclisp, interlisp, zetalisp, lisp machine lisp, and they all could run LISP 1.5 code often without any changes. The commonalities of those lisps got standardized into "common" lisp standard, while each individual lisp still retained their particular quirks. What this meant in practice is that most trivial Lisp code could run on any Lisp system in existence, and most non-trivial Lisp code could run on any Lisp system in existence with only minor modifications. That's why for example a community project to restore ELIZA, written as it was for 1.5, from its published paper could run on any modern common lisp system with only a small support prelude.
So when someone would say "it's lisp code", it meant you could load it up on your Genera machine (zetalisp) OR on your spice lisp (common lisp), tweak a few functions, and it'll work. This is not at all the case with scheme, clojure, or myriad of other "guys! i wrote a lisp last weekend!" projects.
I don't know if modern "CL dorks" just cargo cult the terminology, but there's plenty of old timers, who would've gotten confused if you were to tell them "hey, check out my lisp code" and then threw a scheme code at them.
These days details don't matter, everything is about appearances and cliques, so a lot of people get butthurt if you don't let them call their pet project "a lisp", purely because lisp has a brand recognition, community, history, etc. while "joe's programming language" doesn't.