Hello everyone!
After a long couple weeks of school, I was finally able to work on the code for the automatic hive power updater.
The code was not working the last time I left, but coming back with new ideas and a fresh outlook, I was able to come up with a different solution of how to make the code work. Furthermore in this blog I will be describing how the entire program runs in case you wanted to update it.
Github
All the code is posted on Github. Use this link to clone it: repo,
or go to Thomas-J-Kidd / HivePowerTracker on Github to find my project!
How the code works
This will be broken into a couple of different parts since there are 3 different main sections to this program.
- Data collection
- Sending the email
- Continuing this process over and over in harmony for multiple users
All of these 3 different parts have to work together to ensure that the program runs smoothly. To collect the data we will scrape data from hivestats.io and further do some calculations in our code to parse the html code correctly. Then based off that data, we need to send an email, which we will do through a custom made gmail account. Finally we want to make sure that the above mentioned processes run smoothly forever for as many users as we want! Having multiple users is something I still need to better implement, but we will get there in a later tutorial.
More detail
The program will run by first scraping data from hivestats.io, then we will do a calculation to see how much time remains until the users hive power is full in minutes. Ideally when your hive power is full you will get an email saying its full. Now some users might want to get notifications when their hive power is 95%. In order to combat that we ask the user for a thresh hold parameter. This will let us send emails to users when they truly want them to. Furthermore some users get bloated with emails so we want to ensure that they notice our email. In order to do this we ask them to give us a resend time. This will act as our timer to send new emails if their hivepower is still full. Please see the following diagram below to better understand what is going on as I know it can appear quite confusing.
We need to know several different variables from the users before we start even coding. These variables are the following:
- username
- email address
- thresh hold
- resend time
Thresh hold an resend time are described in the above diagram.
To follow the project fully and understand how to work it in and out, please look up other resources to install certain libraries. In this tutorial I will be only covering how my code works. Not how to write it and how to install libraries and dependencies. All installations are fairly self explanatory and are widely available across the internet.
Part 1 | Data collection
As mentioned above lets see how to collect some data, as the backbone of this project works off of data. There are several different options when it comes to data collection, but for our purposes I found selenium to be the easiest to work with. Selenium can be imported into python and will allow us to easily scrape certain xpaths(html codes) from hivestats.io. By doing this we can gather all the data we will ever need for our project easily, as long as hivestats.io does not change their website formatting. For future reliability the most stable way to get user data would be from the blockchain directly but that takes more coding which I do not yet have experience with. In the future expanding to this would be a great idea though to be independent of hivestats.io and run this service on our own.
Now for the coding part:
First off we will be using object oriented programming or OOP for short. This will allow us to easily reuse the code.
#python
def __init__(self, username, email_address, treshold, resend_time):
# getting drivers / webscraping working
options = FirefoxOptions()
options.add_argument("--headless")
self.driver = webdriver.Firefox(executable_path = r'/home/hivepowertracker/HivePowerTracker/Program/geckodriver', options=options) #this is the path where my geckodriver exists
# getting data paramaters accessible for the whole class
self.username = username
self.email_address = email_address
self.treshold = treshold
self.resend_time = resend_time
self.vote_value_percentage = ''
self.time_to_100 = ''
self.effective_hive_power = ''
self.hive_power = ''
self.total_minute_value = 0
self.total_time = 0
# getting user data
# self.get_user_data()
#print(self)
self.check()
as you can see, we first initialize several different values that will help us determine the users username, email address, threshold, and resend time. Furthermore we initialize the gheckodriver which is the backbone for being able to scrape web data. This is our driver that connects to, in our case to firefox, so make sure you have firefox downloaded on your machine!
TIP: You can do this same process using other browsers but then you need different drivers for those browsers.
Once we initialized everything we want to actually collect data. If you are following along with the code posted in the Github, then you will see the data collection function next in the update_mod.py file. In this function we actually scrape the web data from hivestats.io/username
#python
def get_user_data(self):
hivestats_url = 'https://hivestats.io/@' + self.username
self.driver.get(hivestats_url)
time.sleep(10)
self.vote_value_percentage = self.driver.find_element(By.XPATH,'/html/body/div/div/div[1]/div/div[2]/section[1]/div[2]/div/div/span[1]/span[2]')
self.time_to_100 = self.driver.find_element(By.XPATH,'/html/body/div/div/div[1]/div/div[2]/section[3]/div[2]/table/tbody/tr[8]/td[2]')
self.effective_hive_power = self.driver.find_element(By.XPATH, '/html/body/div/div/div[1]/div/div[2]/section[3]/div[2]/table/tbody/tr[1]/td[2]/span')
self.hive_power = self.driver.find_element(By.XPATH, '/html/body/div/div/div[1]/div/div[2]/section[3]/div[2]/table/tbody/tr[2]/td[2]/span')
# uses get_time function to calculate the remianing time need to wait with the treshold
self.total_time = self.get_time()
return 0
You might see that right before the return statement, we call a new function get_time(). This function, shown below, allows us to parse the html data and understand numerically how much time is left until YOUR hivepower is full!
#python
# gets the number of minutes left for the user to have full hive power again
def get_time(self):
string = str(self.time_to_100.text)
temp = string.replace(" ", "")
self.total_minute_value = 0
if (temp.find("Full") == 0):
self.total_minute_value = 0
# Getting the number of minutes remaining based on the day value
if (temp.find("day") != -1):
day_index = temp.find("day")
day_value = int(temp[day_index-1])
minute_day_value = day_value * 1440
self.total_minute_value += minute_day_value
# Getting the number of minutes remaining based on the hour value
if (temp.find("hour") != -1):
hour_index = temp.find("hour")
if (temp[hour_index-2].isdigit()):
hour_value = int(temp[hour_index-2] + temp[hour_index-1])
minute_hour_value = hour_value * 60
self.total_minute_value += minute_hour_value
elif (temp[hour_index-1].isdigit()):
hour_value = int(temp[hour_index-1])
minute_hour_value = hour_value * 60
self.total_minute_value += minute_hour_value
# Getting the number of minutes remaining based on the minute value
if (temp.find("minute") != -1):
minute_index = temp.find("minute")
if (temp[minute_index-2].isdigit()):
minute_value = int(temp[minute_index-2] + temp[minute_index-1])
self.total_minute_value += minute_value
elif (temp[minute_index-1].isdigit()):
minute_value = int(temp[minute_index-1])
self.total_minute_value += minute_value
return self.total_minute_value-self.treshold
This get_time() function goes through a long string of if/else statements to understand what the wait time is for full hive power. With this function, we have finally built the tools necessary to collect meaningful useful data. Next we must be able to send that data to ourselves or other people!
Part 2 | Sending the email
In this part we will discuss how we use gmail to send updates to ourselves. I know many of you will not like google, but it is the easiest option! So for that reason we went with google, but there are many other ways to do this. You can set up your own email server even if that is something of your interest.
To send an email with google, we must first create a google email, once you have done that you will have to enable developer mode. This will allow you to log in using a username and password only and allow logins from an unknown device. MAKE SURE THAT THIS IS A THROW AWAY EMAIL! Furthermore google will soon require all emails to have 2FA so that will make this a more secure option. Once that is done we can use the built in python smt library and the MIME library to send emails with nice subjects.
#python send_email.py
import smtplib, ssl
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
def send_email(username, email_address):
port = 465 # For SSL
# email service constants
smtp_server = "smtp.gmail.com"
sender_email = "youremail" # Enter your address
password = "youremailpassword"
# messgae
subject = username + " YOUR HIVE POWER IS FULL"
body = "Hello there: " + username + "\n\nYour hive power is full"
message = MIMEMultipart()
message['From'] = sender_email
message['To'] = email_address
message['Subject'] = subject
message.attach(MIMEText(body, 'plain'))
text = message.as_string()
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", port, context=context) as server:
# log in to email
server.login(sender_email, password)
# send email
server.sendmail(sender_email, email_address, text)
print("Sent email to: ", username)
This is the way you send an email using python and a gmail account. If you follow my comments, the code is fairly straight forward. Finally we need to be able to to this over and over for multiple users.
Part 3 | Continuity of the checking process and multi users arrangement
This part is broken down into two parts
- Checking again and again
- Multi user capability
Checking again and again
In order to check again and again we created a check function in updater_mod.py
python #updater_mod.py
def check(self):
print("Checking for user:", self.username)
self.get_user_data()
if (self.total_time <= self.treshold):
send_email.send_email(self.username, self.email_address)
print("waiting for user: ", self.username, " ", self.resend_time, " minutes.")
time.sleep(self.resend_time*60)
self.check()
else:
print("Waiting")
time.sleep(self.total_time*60)
self.check()
This function is a simple recursive function that continues to check and send you emails when you need to. As seen in the flowchart at the beginning of this article, you will see the logic behind this function. This will continue this process of checking and sending emails ideally forever.
Multi User Functionality
To ensure we can do this for many different users, we used the built in multiprocessing library in python.
#python getter.py
import updater_mod
import multiprocessing
#100%_hivepower_0%
# object armoredbanana
def user_1():
user1 = updater_mod.Hive_power_tracker(username = "armoredbanana", email_address = "thomas.kidd@okstate.edu", treshold = 0, resend_time = 60) # treshold when hive power is 100%, resends email when full after 6 hours
# object trostparadox
def user_2():
user2 = updater_mod.Hive_power_tracker(username = "trostparadox", email_address = "trostparadox@gmail.com", treshold = 216, resend_time = 120) # treshold when hive power is 97%, resends email when full after 2 hours
# object bhanutejap
def user_3():
user3 = updater_mod.Hive_power_tracker(username = "bhanutejap", email_address = "bhanutejap20@gmail.com", treshold = 0, resend_time = 1440) # treshold when hive power is 100%, resends email when full after 24 hours
process_1 = multiprocessing.Process(target=user_1)
process_2 = multiprocessing.Process(target=user_2)
process_3 = multiprocessing.Process(target=user_3)
process_1.start()
process_2.start()
process_3.start()
Here we create 3 different processes for 3 different users. This will allow this process to continue with as many users as we like, as long as the computer is capable of running continuously with enough resources. We create a function for the creating of each users object. This allows us to call each function during a process.
Conclusion
In this program we are able to collect data from all of the users we specify on hive, and if we know their email, we can send them an email. If you want to send text messages, or other kinds of notifications, there are plenty of tutorials out there to do so. Please feel free to let me know about any bugs and I will try to fix them!
Thank you for @trostparadox for teaching the class! It was great!
Thank you for reading and I hope you have a great week!