In this project, you will learn the basics of Apache Cassandra, a popular open-source distributed NoSQL database. A three-node Cassandra cluster is integrated into our shopping marketplace to serve listing views and completed-order history, while Postgres still owns account credentials and balances, as well as listing and order states.
While production Cassandra deployment will use multiple physical servers, we use Docker Compose to create a Cassandra cluster via container orchestration for development and testing.
Clone the repository https://github.com/wngjia/ece573-prj04.git that holds scripts and source code for Project 4. Similar to previous projects, please create a new VM by importing our VM Appliance, make a clone of the repository (or fork it first), and provision the instance with the script setup_vm.sh.
This project no longer depends on your completed Project 2. Optionally, you may incorporate your updates from Project 3 by editing the Project 3 TODO sections in 'marketplace-service/main.go' and 'marketplace-service/authorization.go'. The provided workflows use AUTH_ENABLED=false that permits access to account and history without authorization.
Briefly review our README.md, either in its textual form, or directly on GitHub as a web page. Execute the script run_baseline.sh from the project directory to run the starter baseline:
ubuntu@ubuntu24:~/ece573-prj04$ ./run_baseline.sh ... Starter baseline completed. Responses are saved in out/. Cassandra account and listing views were checked after marketplace restart. The services are stopped; the Postgres volume and signing keys are preserved.We organize the scripts in a similar way as previous projects. The baseline uses these scripts to reset the Postgres and the Cassandra databases, start the services, register accounts and grant credit, list and order an item, and finally restart the marketplace service to make further checks.
Open 'docker-compose.yml' to understand how the Cassandra cluster is integrated into our shopping marketplace. The cluster includes three nodes cassandra-1, cassandra-2, and cassandra-3. Environment variables CASSANDRA_CLUSTER_NAME and CASSANDRA_SEEDS define the cluster and help nodes discover each other. While the node settings are shared via 'extends', a separate data volume is used for each node.
Manually restart the services by executing 'scripts/reset.sh' and 'scripts/start.sh' so we can explore the Cassandra cluster further.
ubuntu@ubuntu24:~/ece573-prj04$ scripts/reset.sh ... Signing keys and admin password rotated. Services are stopped. Run ./scripts/start.sh to use the new secrets. Postgres and Cassandra state reset. Admin password is in secrets/admin-password. ubuntu@ubuntu24:~/ece573-prj04$ scripts/start.sh Starting Postgres, the application services, and 3 Cassandra nodes. This may take a few minutes. ... Waiting for the marketplace and login services to become ready... Marketplace service is ready at http://127.0.0.1:8080. Login service is ready at http://127.0.0.1:8081.
Instead of use docker ps to verify that all containers are running, we can use docker compose ps to check the status of the containers for this cluster.
ubuntu@ubuntu24:~/ece573-prj04$ docker compose ps NAME IMAGE ... SERVICE ... STATUS ... ece573-prj04-cassandra-1-1 cassandra:4.1.7 ... cassandra-1 ... Up 3 minutes ... ece573-prj04-cassandra-2-1 cassandra:4.1.7 ... cassandra-2 ... Up 3 minutes ... ece573-prj04-cassandra-3-1 cassandra:4.1.7 ... cassandra-3 ... Up 3 minutes ... ...Pay attention to NAME, IMAGE, SERVICE, and STATUS columns for this project. If anything went wrong, use docker compose logs with SERVICE to troubleshoot the issue.
ubuntu@ubuntu24:~/ece573-prj04$ docker compose logs cassandra-1 ... cassandra-1-1 | INFO [NonPeriodicTasks:1] ... All sessions completedIf the output shows a lot of error messages talking about Java, don't panic as the Cassandra cluster takes time to initialize and you will need to wait a few minutes to run the command again, until the summary of the cluster appears.
As a distributed database system, one Cassandra program is running on each physical server (container for our cluster) to manage the resources locally, and they communicate with each other via networking. To manage the database system consisting of these programs, Cassandra provides two tools, nodetool to access the statistics of the Cassandra programs, and cqlsh to manage the databases and the data via the Cassandra Query Language (CQL), which is very similar to SQL. For simplicity, we don't install these two tools in our VM but use them directly from the Cassandra containers. Use docker compose exec to execute a program in a container. For example, docker compose exec cassandra-1 nodetool status executes nodetool status that shows a summary of the cluster in the container of cassandra-1.
ubuntu@ubuntu24:~/ece573-prj04$ docker compose exec cassandra-1 nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.21.0.4 174.11 KiB 16 100.0% 5327fb23-95ba-4f6d-a130-1b5e26b7b3be rack1 UN 172.21.0.2 156.62 KiB 16 100.0% d65cbc10-9f7a-4fb2-bf61-eedf88cbd5e3 rack1 UN 172.21.0.3 168.84 KiB 16 100.0% 4d1d3b07-3c8e-4825-8079-91d95b150b72 rack1
Start cqlsh in a similar way.
ubuntu@ubuntu24:~/ece573-prj04$ docker compose exec cassandra-1 cqlsh Connected to ece573-shopping-prj04 at 127.0.0.1:9042 [cqlsh 6.1.0 | Cassandra 4.1.7 | CQL spec 3.4.6 | Native protocol v5] Use HELP for help. cqlsh> exitThe output shows that our Cassandra cluster is indeed running because it mentions the cluster name ece573-shopping-prj04 that we have set up in 'docker-compose.yml'. Type exit to exit it. You will need to learn CQL first to make full use it. Feel free to explore further by reading CQL tutorials and documentations online.
Cassandra supports a lot of programming languages, including Go, via CQL and a set of APIs. For Project 4, we provide the history package in marketplace-service to demonstrate how to create keyspaces and tables, and how to write data to and read data from a Cassandra cluster in Go.
Take a look at 'marketplace-service/history/history.go'. This file includes functions to connect to the Cassandra cluster with the given consistency and seed, and to create the keyspace marketplace_history requiring 3 replicas in 'datacenter1' as well as four tables. Among the four tables, both purchases_by_buyer and sales_by_seller contain exactly the same information for a completed order but with different keys to facilitate different queries: purchases_by_buyer is partitioned by buyer ID so queries by buyer as purchases can be queried within one partition, and sales_by_seller is partitioned by seller ID so queries by seller as sales can also be queried within one partition. Both tables use 'order_day', 'ordered_at', and 'order_id' as the clustering key that further support date/time ranged queries. The other two tables listings_by_seller and available_listings tables support different ways to access the listings, either by seller ID or as a whole for unsold items.
The functions provided by the other two files in 'marketplace-service/history' are used by the RESTful handlers in 'marketplace-service/handlers.go'. Both the Postgres tables and the Cassandra tables are updated when listings are created and ordered via 'POST /listings' and 'POST /orders'. After Postgres commits a listing or order, the handler writes the updates to Cassandra. Note that if the Cassandra updates fail, the handler will log the failure but still returns HTTP 201 for the committed Postgres operation. On the other hand, for requests to access accounts and listings via 'GET /accounts/{account_id}' and 'GET /listings', most records are obtained from Cassandra tables. In other words, while Postgres still stores the states for listings and orders, Cassandra provides materialized read views to reduce loads on Postgres, but may disagree with Postgres or with each other due to failures and its own consistency guarantees.
According to the CAP theorem, a distributed database like Cassandra must choose one from consistency and availability during a network partition. Cassandra allows to trade-off consistency with availability by tunable consistency where each write and read could choose a different consistency level. For this project we will experiment with the combinations of three consistency levels: ONE for one replica to confirm, ALL for all replicas to confirm, and QUORUM for more than half of the replicas to confirm. A write followed by a read at ONE/ONE can return an older view because the read need not contact the replica that acknowledged the write. On the other hand, all of QUORUM/QUORUM, ALL/ONE, and ONE/ALL overlap in at least one replica to provide consistency guarantee on the same table row when both operations succeed. However, node availability and application behavior still matter and will require your further investigations.
For your convenience, we provide the script 'run_experiment.sh' to streamline the experimental process. This script takes three parameters: the first is the consistency level applies to Cassandra writes, the second is the consistency level applies to Cassandra reads, and the third specifies the number of available Cassandra nodes at start-up. It will reset the data for each run, waits for marketplace readiness, and creates five listings and five orders. Upon completion, it saves only the HTTP status and formatted JSON body of its final 'GET /accounts/Alice' under 'out/experiment-*.txt', and leaves the services running for your inspections.
Start your first experiment with QUORUM/QUORUM and all nodes available.
ubuntu@ubuntu24:~/ece573-prj04$ ./run_experiment.sh QUORUM QUORUM 3 ... Saved final query to out/experiment-write-quorum-read-quorum-nodes-3.txt Services remain running for additional inspection; the next run resets them.Read the file 'out/experiment-write-quorum-read-quorum-nodes-3.txt'. You should be able to see Alice's records of 5 sold listings, 5 sales, no purchase, and a current balance of 50. At this point, since the services remain running, you can view the logs of the services as we have mentioned in Section II, e.g. for 'marketplace-service':
ubuntu@ubuntu24:~/ece573-prj04$ docker compose logs marketplace-service ... marketplace-service-1 | ... INFO connected to Cassandra marketplace-service-1 | ... INFO Cassandra history schema ready marketplace-service-1 | ... WARN authentication disabled; use only for course development marketplace-service-1 | ... INFO marketplace service listening on :8080 marketplace-service-1 | ... INFO listing listing-1 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO listing listing-2 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO listing listing-3 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO listing listing-4 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO listing listing-5 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO order order-1 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO order order-2 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO order order-3 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO order order-4 committed in Postgres and projected to Cassandra marketplace-service-1 | ... INFO order order-5 committed in Postgres and projected to Cassandra
You will need to experiment with four more cases as listed below that all use two available nodes:
Prepare a project report in .doc/.docx or .pdf format and submit it to Canvas before the deadline. Your project report should include the following for a total of 20 points: