Myself and fellow students at Stanford have been exploring the feasibility of custom graphics pipelines for the past year or two. This work has resulted in a research system called GRAMPS. GRAMPS generalizes ideas from modern graphics pipelines by permitting application programmers to create arbitrary computation graphs (not just pipelines) that contain programmable and fixed-function stages that exchange data via explicitly named queues. The GRAMPS abstractions anticipate high-throughput implementations that leverage a combination of CPU and GPU-like processing cores as well as fixed-function units.
I have recently received a number of requests asking about this work. To those interested, just last week we shipped off a final copy of our paper, entitled "GRAMPS: A Programming Model for Graphics Pipelines" to ACM TOG. If all goes well, I'm told it should be appearing in early 2009. Until then, you can find an electronic copy of the submitted draft here.