Pedestal and w3a walkthrough: Build a CRUD app with Clojure
15 Feb 2016This is a walkthrough on how to build a complete user facing web application with Clojure on top of Pedestal, with the help of the w3a library. I have made various attempts at building web applications with Pedestal and/or Clojure before. (such as here, here and here) The current version is good and useful enough to make a guide for and hopefully serve as a starting point for other Clojure web applications.
For me a complete user facing web application should have a database, templating, forms, authentication and authorization. And for development I like to get something I can click through as soon as possible, as well as having a dynamic development process from the REPL.
The example app is running on Heroku at https://snippetlist.herokuapp.com/. The code is on GitHub at https://github.com/thegeez/snippetlist. Snippetlist is a small CRUD application where users can create and edit little code snippets.
This walkthrough contains the following parts:
- Getting started
- Pedestal basics
- w3a basics
- Using a database
- Context, context everywhere
- The common HTML Flow
- Forms
- Templates with Enlive
- Testing
- Running on Heroku
Getting started
If you want to follow along by reading and playing with the code, you should follow these instructions:
$ git clone https://github.com/thegeez/snippetlist.git $ cd snippetlist $ lein repl => (go)Go to http://localhost:8080 and you should see a simple page with 4 links on it. This development setup follows Stuart Sierra's Reloaded workflow pattern together with Component.
Pedestal basics
Pedestal is a Clojure web framework. The main components in Pedestal are interceptors (which are similar to Ring middleware) and the routing table. When you develop a web application with Pedestal you will mostly be working on the routing table and writing interceptors.
The index page on http://localhost:8080 is defined in src/clj/net/thegeez/snippetlist/service.clj:19-43
19 (def home 20 (interceptor/interceptor 21 {:enter (fn [context] 22 (merge context 23 {:response 24 {:status 200 25 :headers {"Content-Type" "text/html"} 26 :body (str "Hello world, snippetlist<br/>" 27 "<a href=\"" (link/link context :users/index) "\">/users</a><br/>" 28 "<a href=\"" (link/link context :snippets/index) "\">/snippets</a><br/>" 29 "<a href=\"" (link/link context :auth/login) "\">/login</a><br/>" 30 "<a href=\"" (link/link context :home) "\">/</a>")}}))})) 31 32 (defroutes 33 routes 34 [[["/" 35 ^:interceptors [auth/with-auth 36 (breadcrumb/add-breadcrumb "Home" :home) 37 (html/for-html 404 (constantly "Not found")) 38 39 (edn/for-edn (fn [context] 40 {:data (get-in context [:response :data]) 41 :breadcrumbs (:breadcrumbs context)}))] 42 43 {:get [:home home]} ...
There are 3 things to note here. First there is the routing table called 'routes' defined by 'defroutes'. It says that a GET request to the url "/" should be handled by the 'home' interceptor. The home interceptor returns a very crude HTML response with 4 links in it. The routing table also shows 4 other interceptors that are in the interceptor chain before the home interceptor, but these are not important for now. The routing table also shows that it supports bi-directional routing and link generation. The "/" route is called :home. This same route is referred to in the home interceptor to create a link that points to http://localhost:8080/ at line 30.
Bi-directional routing and link generation is nice because with it you don't need to create links by concatenating strings.
When you change something in the home interceptor, for instance by replacing "Hello world" with something else, you will see it in the browser when you refresh the page at http://localhost:8080.
w3a basics
Snippetlist is a simple CRUD application. CRUD stands for Create-Read-Update-Delete. The simplest part of that is the read part. An example of a page that is only the read part is the page that shows all the users in the application. This page is at http://localhost:8080/users. This page shows all the users in the system, which will be only the users Amy and Bob that have been inserted into the database as fixtures. The page lists the users and the links to all the code snippets they have in the database. The users page also has some basic HTML markup in the form of a page header, a box in the top right corner that show the currently logged in user or a link to login, and a breadcrumb bar with links to other parts of the application.
The users page uses interceptors and other helpers from the w3a library to build a user facing web application on top of Pedestal. The w3a library aims to help to get an application to a version that you can click through quickly. The idea is that when you have all the data and functionality available, you can then add the markup and other fancier features at a later time.
To make an application you can click through you need links. For the application to be interesting you also need data. A piece of data with links with it can be called a resource. On the users page there are two user resources: the user Amy and Bob. The user resources consist of data from the database and links, which are mostly generated based on the data. Links on a resource aren't just for navigation, they can also tell you something about a resource. For example, maybe a resource has a link to an edit page when you are logged in as that user. (In the snippetlist application you can see that in action on a particular snippet page, if you are also logged in as either Amy or Bob.)
Another helper to make the application click throughable quickly is the breadcrumb bar. The breadcrumb bar contains a link to the Home page and the current Users page. If you click on the user page for Amy, you'll see 3 links in the breadcrumb bar, including a link back to the users page. The breadcrumb bar is build by adding the breadcrumb/add-breadcrumb interceptors to the Pedestal routing table.
Lets look at how you create this page with w3a:
src/clj/net/thegeez/snippetlist/service.clj:55-60
55 ["/users" 56 ^:interceptors [(breadcrumb/add-breadcrumb "Users" :users/index)] 57 {:get 58 [:users/index 59 ^:interceptors [(html/for-html 200 users.view/html-render-index)] 60 users/index]}
src/clj/net/thegeez/snippetlist/users.clj:7-28
7 (defn user-resource [context data] 8 (let [{:keys [id]} data] 9 (-> data 10 (dissoc :id) 11 (assoc-in [:links :self] (link/link context :users/show :params {:id id})) 12 (update-in [:snippets] (fn [snippets] 13 (map #(link/link context :snippets/show :params {:id (:id %)}) snippets)))))) 14 15 (defn get-users [context] 16 (->> (jdbc/query (:database context) ["SELECT id, username, created_at, updated_at FROM users"]) 17 (map (fn [user] 18 (assoc user :snippets 19 (jdbc/query (:database context) ["SELECT id FROM snippets WHERE owner=?" (:id user)])))) 20 (map (partial user-resource context)))) 21 22 (def index 23 (interceptor/interceptor 24 {:enter (fn [context] 25 (merge context 26 {:response 27 {:status 200 28 :data {:users (get-users context)}}}))}))
The users page shows the users in the application. This is all the users/index interceptor returns. This is done by getting the users from the database with get-users and then adding links to the users to make them resources with user-resource. The HTML layout is added by the html/for-html interceptor, defined in the routing table. If you remove the html/for-html interceptor then this code will render an empty page. This is because the response will not have the proper headers added to the response. (These headers are defined explicitly in the home interceptor we saw earlier.) Of course during development you might not have a template function for every page that you're building yet. A handy temporary tool is to use the html/for-html interceptor with a helper template function.
["/users" ^:interceptors [(breadcrumb/add-breadcrumb "Users" :users/index)] {:get [:users/index ^:interceptors [(html/for-html 200 ;; this replaces: users.view/html-render-index (fn [context] (-> context (select-keys [:breadcrumbs :response]) html/edn->html)))] users/index]}
This will render all the resources and links in a simple pretty printed text layout.
For the /users page the users/index interceptor is very simple. But it is a goal of w3a to define the functionality of an application in a single place, and therefore to have the gathering of data and link generation together. So you are discouraged from creating links in a templating function. It is better to add a link to a resource in the interceptor for the route and to use this data in the templating function.
Using a database
In a web application you probably need a place to save the data, w3a also provides some help for that. Of course there are many different ways to store data, and even more types of databases. w3a has helpers to work with SQL databases, but it does not impose a requirement to use them and you can use however many databases of whatever type you like. The basics in w3a are to get you started and to support getting an application running on Heroku, which happens to provide a free PostgreSQL database.
When you use a database you'll quickly need a way to migrate databases to whatever new version you create. It is also handy to have a way to create fixtures, which is data you can use while developing or testing the application. w3a provides components for migrations, fixtures and database access to an SQL database.
The w3a component supports uses an embedded database when passing a Derby connection string, and PostgreSQL when passed a postgres url. Using an embedded database can be useful for testing and development. The connection strings are in the jdbc connection string format. The database configuration that Heroku uses also works with the database component.
The components all work with the Component library. To see how the database components are setup and configured, see the files dev/user.clj and src/clj/net/thegeez/snippetlist/core.clj. This should also be the starting point to see what your own components should look like if you add a different database.
As could be seen in the users page example, snippetlist uses clojure.java.jdbc to access the database. Clojure.java.jdbc is great for SQL query access and transforms all data into Clojure datastructures. I prefer this over using an ORM-ish DSL, but you can easily use whatever database access library you like instead. As can be seen in the code for the users page, a reference to the database to use with clojure.java.jdbc is part of the context that is passed to every interceptor.
Context, context everywhere
In Pedestal everything goes through interceptors, which take the context as an argument. This context consists of the request, response and other user defined content. Also the references to the database and other components are available through the context.
Apart from passing the context to all the interceptors, the context is also passed to almost every helper function in the w3a library. You could argue that this is not much better than global objects or injected bindings everywhere, but I prefer it as I always end up needing just a little part of the context somewhere deep down in the composition. The biggest example of this is the creation of links with the w3a library taking the context as an argument. In Pedestal the context is not passed as an argument to the link building fuction, instead the routing table is available there as a binding.
The common HTML flow
The w3a library is meant to specifically support user facing web applications. This means that the end user will use the application through a browser. It would also be nice to support more API-like access. Perhaps users won't use a command line to access the application, but maybe some functionality needs to be available via JavaScript/AJAX access.
Sadly, the common behavior a user expects from a webpage in a browser and the common way to create API access are not always aligned. For instance when a users creates a new snippet in the snippetlist application, the users expects to be automatically redirected to the newly created snippet page. But for API access perhaps returning a '201 Created' response is more appropriate.
In w3a all the code is written as if the data will be accessed in an API style. So for instance the interceptors that creates a new snippet returns:
src/clj/net/thegeez/snippetlist/snippets.clj:160-163
160 {:response 161 {:status 201 162 :headers {"Location" location} 163 :flash {:info "Snippet inserted"}}}
When this code is accessed through a browser requesting HTML, then w3a will transform the response into a '303 See other' redirect instead. This transformation is done for '201 Created', '204 No content' (after a successful update), '401 Not authenticated' (redirects to login) and '403 Not authorized' (redirects to login) responses. Also, when an update to a resource fails, for instance when a form doesn't validate, the response code 422 will be a 200 for browsers instead (with the page rendered again with some error messages).
A redirect does not inform the user about what happened. A common way to tell a user what just happened is through a 'flash' message. This is supported in w3a by putting a :flash key on the response. When a user inserts a snippet, the user is redirected to the list of snippets and the user sees a blue box with the flash message saying: "Snippet inserted". Flash messages are only shown once. When the page is refreshed the flash message will be gone.
Forms
For users to create or update something in a CRUD application in the browser, you will need forms. w3a provides helpers to work with forms.
If you login as Amy on the snippetlist application you can create and edit snippets through a form. Login first at http://localhost:8080/login and then go to http://localhost:8080/snippets to select a snippet to edit or to create a new one.
src/clj/net/thegeez/snippetlist/snippets.clj:86-107
86 (def snippet-form 87 [{:id :owner_username 88 :label "Owner" 89 :type :static} 90 {:id :code 91 :label "Code" 92 :type :string 93 :validator (fn [code] 94 (when (not (seq code)) 95 "Code can't be empty"))} 96 {:id :quality 97 :label "Quality" 98 :type :checkbox 99 :render :snippet/quality-field 100 :render/options {:fast "Fast" 101 :good "Good" 102 :cheap "Cheap"} 103 :coerce (fn [value] 104 (set (keys value))) 105 :validator (fn [quality] 106 (when (= (count quality) 3) 107 "Sorry, you can only select up to 2 qualities"))}])
The form definition above serves two purposes: it can be used to generate an HTML form from (see src/clj/net/thegeez/snippetlist/snippets/view.clj:108-112) and it can be used to parse and validate incoming request from pages where the page was displayed (see src/clj/net/thegeez/snippetlist/service.clj:79). It also support rendering a form with error messages in case the validation failed.
When form validation fails in w3a the data from the form parsing will have an :error key in it. In w3a it is common to render a form that failed validation again with the 422 status code. (see src/clj/net/thegeez/snippetlist/snippets.clj:126-134). If you have more elaborate validations, such as checking that something doesn't already exist in the database or checking that two input fields make sense together, you can do so in the route handler. This is also the place you could add extra error messages to the form data when you want to re-render to form again.
Currently w3a support only a couple of types of form inputs. But this system is extensible through the use of multimethods. (see clj/net/thegeez/snippetlist/snippets/view.clj:78)
Templates with Enlive
HTML pages in w3a are generated with Enlive. Enlive combines plain HTML pages with data and merges the two together, without having to add new syntax into the HTML pages and by passing along all the data explicitly.
Enlive is a bit different from other templating systems. Its selector based approach might take a while to get used to, but it is worth it to learn because it is also useful for testing with the kerodon library. You can use other templating libraries with w3a as well. The w3a form helpers return Hiccup compatible structures.
Testing
As said before, w3a applications are meant to be used by a user through a browser as well as through API access. To test whether the application does what you want it to, the kerodon and peridot libraries can be used.
With kerodon you can test the HTML flow of the application. As you would when using a browser, you can follow links, fill in forms and press buttons. For each page that gets rendered, you can check if the page has all the elements you expect it to have. kerodon doesn't launch an actual browser to run the tests, it works by making request to the application and parsing the responses. For inspecting the responses kerodon uses Enlive to find elements in the returned HTML.
A piece of the kerodon test for snippetlist from test/net/thegeez/snippetlist/html.clj:11-24
11 (deftest snippet 12 (-> (k/session (test-core/ring-handler)) 13 (p/header "Accept" "text/html") 14 (k/visit "/") 15 (follow "/snippets") 16 (k/within [:#content] 17 (t/has (t/link? "http://testhost:-1/snippets/new")) 18 (t/has (t/link? "http://testhost:-1/snippets/1")) 19 (t/has (t/link? "http://testhost:-1/snippets/2"))) 20 (follow "http://testhost:-1/snippets/new") 21 (k/follow-redirect) 22 (at? "/login") 23 (k/within [:div#flash-info] 24 (t/has (t/text? "You need to be logged in for that action.")))
For testing the API access to a w3a application you can use peridot (which is used internally by kerodon as well). You can specify the requests to the application and test the responses again, which are now EDN data instead of HTML forms and pages.
A piece of the peridot test for snippetlist from test/net/thegeez/snippetlist/edn.clj:43-58
43 (-> res 44 (p/request create 45 :request-method :post 46 :params ^:edn 47 {:snippet 48 {:code nil 49 :quality 50 {:fast true 51 :cheap true 52 :good true}}}) 53 ((fn [res] 54 (is (= (-> res :response :status) 422)) 55 (is (= (get-in res [:response :edn :data :snippet :errors]) 56 {:code ["Code can't be empty"] 57 :quality ["Sorry, you can only select up to 2 qualities"]})) 58 res))
Both kerodon and peridot are designed to work with the -> (thread first macro). With this every step is either an action to go from the current browsing state to the next one or a check to test something. If you get stuck with either kerodon or peridot, a useful trick is to insert this code somewhere:
(-> ..some actions & checks... ((fn [state] (println "The state is now: " (pr-str state)) (println "With kerodon the Enlive selection is now:" (:enlive state)) state)) .. the next actions & checks ...)
Both kerodon and peridot work with clojure.test so they can both easily be used in a test suite and they also support using testing fixtures.
Running on Heroku
The snippetlist repo contains everything required to run on Heroku. Heroku is a cloud application hosting provider, with a free option for hobby projects. The snippetlist requires a database, which can be added to the applicaiton with:
heroku addons:create heroku-postgresql:hobby-dev
You'll also need to configure the cookie key:
heroku config:set COOKIE_KEY="SOME16CHARSTRING"
After the database has been added to the Heroku application, you'll need to run the migrations and fixtures to fill the database with the required tables and some test data to play with.
Migrate the database with:
heroku run lein run -m net.thegeez.snippetlist.main/database-migrate
Insert the fixtures with:
heroku run lein run -m net.thegeez.snippetlist.main/database-fixtures