SpiceRack

Project Description

SPICE data is used by scientists and engineers to manage spacecraft and the instruments used on the spacecrafts. The problem is that whenever a team of people download a set of SPICE data to use and manipulate it can be a difficult task to manage and keep all of the data in-sync. The goal of this project is to create a microservice using Docker. Inside the Docker container will be a ReSTful API which is a manager that has the ability to relay differences in client data with each other. We will also be developing a peer-to-peer distributed network that makes synchronizing SPICE data between local directories possible, using the manager and file transfer protocols (FTP/SFTP) to transfer the data over the network. After developed this piece of software will be used to make another microservice that will be used by the USGS to take advantage of our manager to make a ReSTful SPICE interaction. Utilizing the manager and ReSTful SPICE will save researchers, scientists, and engineers a lot of time configuring SPICE data that could spent on their research or their work.

High Level Requirements

Synchronizing SPICE data is a hard task to accomplish, especially when working with multiple people that may alter, remove, or add files. We are going to implement a few things to make this possible:

Microservice to sync client’s SPICE data using Docker
A ReSTful manager that monitors directories and what is inside them.
Peer-to-peer distributed network to transfer the data to keep the directories synced and up to date.

Architecture

The architecture as illustrated by the drawing consists of two general components the manager, and the orchestrater. The client will connect to our ReSTful API i.e. the manager, which will then retrieve the SPICE data from the specified directory. The manager will then be able to communicate with other managers to be able to tell what files need to updated accordingly. After the manager receives this information the orchestrator will then issue the transfer process of updating the client that needs the files through the peer-to-peer network. Our design paradigm is mostly going to be Agile, meaning we will break the implementation into smaller tasks and distribute them to the teammates.

Technologies

The technologies used will be a combination of a framework and a few libraries. We will be developing in Python because of the extensive frameworks and libraries offered to us. To be able to assign tasks we will also be using GitHub to manage our software on a repository managed by the USGS. We will also be utilizing Docker during development so our software can be used a microservice in the future. The primary framework will be python Flask, and the libraries will be: pandas, pyp2p, farmhash, and hashlib. The libraries are not concrete as we are still looking into the best way to accomplish this (we will decide more during implementation). These are our initial thoughts, and may change in the coming weeks. For now let’s take a look further into these technologies:

Docker: Docker allows you to create mini-VM’s called containers that allow you to store all the libraries, codebase, system calls, and much more in a confined space. This container will allow our software to be ran as a microservice, meaning anyone who has docker can download our image, create their own container and run it using Docker making our software much more portable.
Python Flask(): Allows an easy way to develop a ReSTful API that will suit our clients needs. Flask abstracts a lot of the annoying and nitty gritty parts of a ReSTful service allowing us to focus more on the features and how the ReSTful is setup and functions.
Pandas/pyfarmhash: Pandas is essentially an in memory database which will allow us to store the files in a dataframe hash it using pyarmhash and then be able to check the difference in files and see which files need to updated or transferred. We will be testing this against a checksum and determine what is faster or more efficient.
Pyp2p: This is a library for building a peer-to-peer network which will be used to transfer files between clients, we are still looking into this, but this might be able to save us the hassle of building this from the ground up.
Hashlib: Hashlib will allow us to perform an MD5 checksum on the file system which essentially will check to see the difference in bytes and then be able to transfer those bytes (files) over the network to keep the clients in-sync.

Codebase

SpiceRack GitHub Here