Hey there, hope everyone is doing fine! 😃

Today we’re going to tackle the first part of our introductory DevOps tutorial that will be splitted in 4 distinct parts and through which we will learn among other things how to:

• Handle API calls through Python using Santiment API

• How to properly set up a coding project

• How to create and set up a distant cloud server on Digital Ocean

• Create and manage PostgreSQL database on a distant server

• Build-up data tunnel between an API provider and a distant data server

• How to connect a Grafana instance to a cloud hosted database

Phew! 😬

So as you can see we have quite an awful lot to do here. However, it will all be worth it as by the end of this tutorial we will have a basic understanding on how to create a complete data architecture from scratch, so grab a cup of coffee and once you’re ready join me in the first part of this tutorial.

Project Presentation

So first before diving right-in into the technical aspect of this tuto let’s discuss a little bit about what we’re trying to do here 🤔

Basically, the idea of this tutorial is to give us a “grand tour” of the main DevOps components and skills that are needed to create a full data infrastructure from scratch. As such, I thought that a good way to do just that would be to create the whole back-end data architecture supporting a Granafana analytical dashboard focusing on the Ethereum network and using the following metrics:

• HIstorical price

• Volume

• Token circulation

• 24h active addresses

• Velocity

• Market capitalisation

• Top 10 holders holdings

• Github activity

• Social Volume

• Dev activity

• Network Growth

Alright, so now that we have an idea of what we want to do, the next question is how to do it?

Basically, here we’ll use the most simplistic approach in order to just give you a basic sense of how it can be done. As such, we’ll basically use the Santiment API provider in order to get our needed metrics and then store those value in a set of PostgreSQL databases that we’ll host on a Digital Ocean cloud server. Then, we’ll just connect those databases to a Grafana instance and simply build our Grafana dashboard from there.

Data architecture flow

• Step 1 — Our Python scripts stored in our Digital Ocean server make API requests to extract the needed metrics for our dashboard.

• Step 2 — Santiment API return the metrics value that we reformat and store in our custom PostgreSQL databases hosted on Digital Ocean.

• Step 3 — Our Grafana instance connected to those PostgreSQL tables upload the new metrics values and update accordingly our Ethereum analytical dashboard.

Step 1 Requesting data from Santiment API

Alright so first thing first before thinking about the rest of our architecture we need to create some code scripts in order to extract our needed metrics from the Santiment API provider.

Here we basically have two main possibilities to make our API requests either using curl command inside bash scripts or use the python library developed by Santiment. So even though we could dive right into bash scripting, given that the idea of this tutorial is to keep things simple we’ll instead use the handy python library at our disposal here.

(For those interested, I’ll also do a bonus part of this tutorial where we’ll create the same architecture but using Bash so stay tuned if you’re interested in that 😉)

So first thing first, you’ll need a python IDE, you’re free to use the one you prefer but on my end I’ll use the Spyder IDE coming with the Anaconda distribution so if you don’t have any IDE or want to have the same screen as me just download anaconda from here.

Alright so now that we’re set up, before being able to start coding our python API requests, we’ll need to install the python library developed by Santiment API. So to do that open your terminal and if you’re not using any virtual environment copy and paste the following command inside your command line before pressing enter.

Cool, we’re now all set up to start coding and creating our API requests. So go ahead and create a dedicated folder where we’ll store the code for our project. Here is a basic example, but feel free to use whichever template you’re most comfortable with.

From there, let’s open our python IDE and create our first code script that we’ll call function.py, in which we’ll code as functions all the API requests that we need to make to the Santiment API. The idea here being to keep things organised.

Alright so first stop, let’s create a function to request from Santiment the ETH historical price using the guidelines provided by Santiment in their documentation:

As you can see it is pretty straightforward and pretty much speaks for itself. So let’s try it out now in order to check that everything is working.

So first we’ll need to import a few packages among which datetime and sanpy.

Alright, and from there, let’s us do a basic testing of this first function:

And it works as expected, as you can see from the above illustration extracted from the Spyder variable explorer.

Alright, so now that we did the first example, we’re basically going to reiterate the same process for each case and create a function for each of our metrics.

Marketcap Function

Network Growth

⚠️⚠️

In the following cases given that we only want those metric for the current date, there is no need for a start date and an end date and can instead use here a unique date input that we’ll use both as “from_date” and “to_date”.

⚠️⚠️

Current Price

Current Volume

Daily Active Address

Token Circulation

Velocity

Social Volume

Top 10 holders holdings

Dev Activity

Github Activity

⚠️⚠️

Important note: Given that we are using the free Santiment API plan here, for some metrics we don’t have access to the latest value and view ourselves limited to only being able to request at most last month values. So it’s not ideal but given that the goal of this tutorial is just to learn how to create an overall data architecture it is not really critical here.

List of restricted metrics: Circulation, Velocity, Social Volume, Amount in Top holders, Dev activity, Github activity and Network growth.

⚠️⚠️

Alright and that’s it we have now at our disposal all the needed functions to request all our metrics from the Santiment API. So if you want to test it for yourself, here is the full test code:

So congrats, you made it through this first part of our DevOps tutorial! 🥳🥳

Next time, we’ll spin up our cloud server on Digital Ocean, create our PostgreSQL database and write our main code script calling those python functions in order to populate our database so stay tuned and happy coding in the meantime.

Take care 🖖

N.B As per usual full code available here on Cybergen Lab Github repo