Crux: Our final database migration

1.1. The problem

At Avisi Apps, we build apps for Atlassian products such as Jira. We're currently mainly focused on the Atlas CRM app. Atlas CRM has always been available for Jira Cloud instances and has recently become available for Jira Server. Building apps for Jira Cloud is completely different compared to Jira Server. We host the Cloud app ourselves, which gives us total freedom in making architectural decisions. On the other hand, the Server app is hosted by the customer which results in technical constraints.

We use Clojure, ClojureScript, and Datomic for Cloud. We managed to get Clojure and ClojureScript working in OSGI for our Server app, but we had to use Active Objects in our data layer. Active Objects is an ORM layer in Atlassian products, which enables app developers to access and store data within the product. 

Using Active Objects comes with SQL queries - something that we haven't seen in a while. The ease of writing queries with Datalog for Cloud has spoiled us, which resulted in us having a hard time querying data for our Server app. As we continued developing our Server app, we realized that the difference between the data layers was expanding rapidly as writing and reading data for the same Atlas CRM functionality was completely different.

As we made progress with our Server app, we ran our first database migration. We discovered that doing database migrations on Server was a lot harder compared to Cloud. This was mainly due to the lack of documentation and best practices about migrating Active Objects tables.

1.2. Our switch to Crux

Crux is an open-source document database with bitemporal graph queries. It has been available as a public alpha since April 19th 2019. It is similar to Datomic because the graph database and queries are written with the Datalog query language. Our team saw Crux as an opportunity to align the Cloud and Server development and to get rid of issues that we ran into with Active Objects.

I can hear you thinking: "Weren't you restricted to use Active Objects for your Server app?" Well, Crux allows us to continue using Active Objects, here is how... 

Crux uses a data store to write transaction and document logs (at this time, Kafka is the only supported data store). Based on these logs, Crux is able to build indexes that are used for querying data. These indexes are stored within a Key/Value store. Crux currently supports implementing RocksDB and LMDB as Key/Values stores.

To make Crux work for our Server app, we had to write and open-source an Active Objects implementation for the transaction and document log. This implementation writes the Crux transactions and documents into Active Objects tables which make it compatible with Jira Server apps. We were happy to see that it actually worked but we also ran into issues with the RocksDB and LMDB implementations of the Key/Value store. Choosing one of these implementations would cause our out-of-the-box app to not be compatible with our customers' machines. To solve this issue, we wrote and open-sourced a Xodus implementation of the Crux Key/Value store since Xodus is written in pure Java and Kotlin.

1.3. Querying data

We are currently in the process of rewriting our app to work with this Crux implementation. From this point forward, we don't have to perform any Active Objects migrations or SQL queries anymore. As mentioned above, Datalog is the query language that is used to query data in Crux. Datalog is a declarative logic programming language which makes queries very easy to write and understand. So let's get started!

Let's say we are building a webshop with categories and products. The products have the following attributes:

• name
• in-stock-amount
• price

The categories have the following attributes:

• name
• products

As there are no schemas in Crux, we don't have to define these attributes. A document can contain any valid EDN.

(api/submit-tx crux-db
[[:crux.tx/put
 {:crux.db/id :category.id/electronics
 :category/name "Electronics"
 :category/products #{:product.id/laptop :product.id/phone :product.id/tablet}}]
[:crux.tx/put
 {:crux.db/id :category.id/books
 :category/name "Books"
 :category/products #{:product.id/hunger-games :product.id/a-song-of-ice-and-fire}}]
[:crux.tx/put
 {:crux.db/id :product.id/laptop
 :product/name "Laptop"
 :product/price 900
 :product/in-stock-amount 4}]
[:crux.tx/put
 {:crux.db/id :product.id/phone
 :product/name "Phone"
 :product/price 300
 :product/in-stock-amount 8}]
[:crux.tx/put
 {:crux.db/id :product.id/tablet
 :product/name "Tablet"
 :product/price 450
 :product/in-stock-amount 0}]
[:crux.tx/put
 {:crux.db/id :product.id/hunger-games
 :product/name "The Hunger Games"
 :product/price 15
 :product/in-stock-amount 20}]
[:crux.tx/put
 {:crux.db/id :product.id/hunger-games
 :product/name "A Song of Ice and Fire"
 :product/price 12
 :product/in-stock-amount 14}]])

(api/q crux-db
       {:find '[?product-name]
        :where '[[?product :product/in-stock-amount 0]
                 [?product :product/name ?product-name]]})

=> [["Tablet"]]

(api/q crux-db
       {:find '[?product-name ?category-name]
        :where '[[?product :product/in-stock-amount 0]
                 [?product :product/name ?product-name]
                 [?category :category/products ?product]
                 [?category :category/name ?category-name]]})

=> [["Tablet" "Electronics"]]

(api/submit-tx crux-db
               [[:crux.db/id :product.id/phone
                 :product/name "Phone"
                 :product/price 250
                 :product/in-stock-amount 8]])

(api/entity crux-db :product.id/phone)
; Phone has a price of '250'
=> {:crux.db/id :product.id/phone
    :product/name "Phone"
    :product/price 250
    :product/in-stock-amount 8}

(api/history crux-node :product.id/phone)
; There are two versions of the document with id :product.id/phone
=> ({:crux.db/id "c7e66f757f198e08a07a8ea6dfc84bc3ab1c6613",
     :crux.db/content-hash "a95f149636e0a10a78452298e2135791c0203529",
     :crux.db/valid-time #inst "2015-05-18T09:20:27.966-00:00",
     :crux.tx/tx-time #inst "2019-04-15T07:53:56.178-00:00",
     :crux.tx/tx-id 1555314836178}
    {:crux.db/id "c7e66f757f198e08a07a8ea6dfc84bc3ab1c6613",
     :crux.db/content-hash "6ca48d3bf05a16cd8d30e6b466f76d5cc281b561",
     :crux.db/valid-time #inst "2018-05-18T09:18:22.821-00:00",
     :crux.tx/tx-time #inst "2019-04-15T07:53:55.817-00:00",
     :crux.tx/tx-id 1555314835817})

(api/entity crux-db "c7e66f757f198e08a07a8ea6dfc84bc3ab1c6613")
; The older version of Phone has a price of '300'
=> {:crux.db/id :product.id/phone
    :product/name "Phone"
    :product/price 300
    :product/in-stock-amount 8}

1.4. Eviction

The ability to reverse our steps and go back in time has saved us more than once on Cloud with Datomic. Being able to debug what went wrong when something unexpected occurs is an example of when going back in time is extremely powerful. If we choose to migrate our Cloud app to Crux, we will be able to keep this feature as Crux offers it too. Another great advantage that comes with migrating to Crux on Cloud is that eviction of data is much easier compared to Datomic. Due to the new GDPR regulations, we have to make sure that we are able to completely delete customer data with all the history that comes with it.

We plan to align all our apps to have the same Crux data layer in the future. We are currently rewriting our Server app to use Crux.