Intermediate
Learn Python Series):In the previous Learn Python Series episodes, we started to work with JSON files, using serialization/deserialization techniques back and forth between JSON and Python dictionaries. Storing JSON data to files is of course a very convenient way to exchange data between applications and different computers (on the internet) but we can - and we will - take it even a few steps further by working with MongoDB and its official Python driver PyMongo.
Let's dive right in!
MongoDB is - in short - a database engine. If you've experimented with programming here and there, you must have heard of "SQL" (Structured Query Language) which is a language with which you can interact with relational databases, such as for example MySQL. MongoDB functions differently, and its data is structured differently, than is the case with a relational database. MongoDB is a database engine which is oftentimes called of the form NoSQL, with which you store and retrieve data in a completely different way. And there are also several types of "NoSQL" databases, where MongoDB is of the form document datastore.
Okay, so there are differences between SQL / relational databases and NoSQL databases, but which differences, why do those differences matter, and when to use what?
I could write an entire book about trying to (in-depth) answer the above questions, but to keep things short and clear, let's compare some features, strengths and weaknesses, between SQL and NoSQL databases:
SQL:
NoSQL:
PS: as time goes by and database development grows, the core differences between SQL and NoSQL database engines begin to fade away. PostgreSQL for example (an SQL / relational database by nature) nowadays also supports JSON like data storage and retrieval, although currently for that purpose lots of MongoDB specific advantages are still lacking in the PostgreSQL solutions.
As said above, in MongoDB you store JSON-like data. "Under the hood" in MongoDB that data is stored in a binary format, BSON, binary JSON, but from a developer / user perspective just think of the MongoDB data as being JSON without the hassle of having to store and retrieve complete JSON structures to "flat" files. Because it's so flexible, the fun thing is that can very quickly develop applications on top of your MongoDB datasets, without needing to "architect" everything you want in your application beforehand (as would be the case with a SQL setup, requiring you to think about and know about all the ins & outs of your application and therefore data structure before beginning to develop it, otherwise you could get "into trouble" later on).
In MongoDB, you can store all the data in your model within a single "document". For example, in case you'd like to store everything about your own Steem account, you can do so within one single document! Not just your "account_name", "account_creation_date" , "account_about_info" (as you would in an SQL context), but you can add all your article posts, comments, your follows, your wallet, whatever you like within the same one "account" document! And even if you have loads of data stored regarding "account 1", you could have less, more or different data stored about "account 2". All that and you can still use matching (eg. ==) and comparison (<, >) type queries!
And to take things even further, MongoDB scales very well ("horizontally"), is RAM-efficient, you can "shard" data (handy for load balancing) and because the data is so self-contained (document 1 doesn't need to relate to / reference to document 2, therefore it's not "relationally oriented") load-balancing is relatively easy too.
Does it show I'm quite enthusiastic about MongoDB? ;-)
Alright, so much for theory and conceptual MongoDB explanations; let's now try to actually use MongoDB together with Python! The PyMongo distribution is the "official way" to use MongoDB together with Python and it contains serveral tools for interacting with a MongoDB instance via Python.
In order to use MongoDB together with Python, you must of course first install and run MongoDB on your computer system. Since this is the Learn Python Series, the core focus of it is Python, so I suggest you visit https://docs.mongodb.com/manual/administration/install-community/ for installation instructions for either macOS, Linux and Windows regarding the installation of MongoDB Community Edition.
A convenient way to manage your MongoDB databases is by installing and running a MongoDB management console client on your computer system. "Robo 3T" (formerly named "RoboMongo") to be found and downloaded on https://robomongo.org/download is my personal favorite to do just that. It's available for macOS, Linux and Windows as well, allows for connecting to multiple MongoDB databases, has multiple views (you can inspect your MongoDB data in various ways), and it has an interactive MongoDB shell, including auto-completion tools, which is pretty convenient to quickly test some queries. (PS: For php / MySQL backgrounded people: Robo 3T / Robomongo does for you (and more) what you're used to with "phpMyAdmin".)
The easiest way to install and use PyMongo is via:
python -m pip install pymongo
If all goes well, you're now good to go and start using MongoDB together with Python!
In order to connect Python to MongoDB, you first need to import the pymongo package, use the MongoClient class from it and set up the connection. Like so:
import pymongo
client = pymongo.MongoClient('mongodb://localhost:27017')
This means you want to connect to the default host MongoDB is running on (being localhost, on your computer system) and you'll be using the default port 27017. The above line could have also be written as:
client = pymongo.MongoClient('localhost', 27017)
or even as
client = pymongo.MongoClient()
... which will all, in most situations, do the exact same thing, and mostly depends on how explicit you want to be in defining how to establish your connection.
What I've called "MongoDB" is actually a "MongoDB server" (which could run on your personal computer, it's then still called a MongoDB server!), and a MongoDB server can run multiple MongDB databases. In your Python code, you must specify which database you want to interact with.
To begin with, you don't have any running and the fun thing with PyMongo that in case the connection with the MongoDB server is established and it cannot find the database you want to interact with, it creates it on the fly. Let's instantiate a new database object db and connect to its (currently non-existent) database called test_mongo like so:
db = client.test_mongo
insert_one()Ok, let's pretend, for now, to begin with, we are going to create a Utopian / Steem / Steemit like database, and in it could be multiple collections of documents, but let's start with the first collection called accounts and in it we're going to store one document (containing account data) in the following (JSON-like) format:
{
"account": "scipio",
"slogan": "Does it matter who's right, or who's left?"
}
We must now first specify the collection (accounts) we want to store this document in, within the selected database (test_mongo), and then we can insert this one document inside the accounts collection using the insret_one() method. Like this:
coll = db.accounts
account_data = {
"account": "scipio",
"slogan": "Does it matter who's right, or who's left?"
}
result = coll.insert_one(account_data)
If we then refresh and look into our Robo 3T MongoDB management interface, we get the following result:
Cool! We did it! :-)
As you can see, the inserted document automatically received an ObjectId value associated to the _id key (we didn't need to manually insert just now), and we can return (or print) its value like so:
print('The ObjectId: {}'.format(result.inserted_id))
The objectID: 5ae46285dd58330cd666056f
insert_many()If you need to insert more than one document to a collection, you could use insert_one() multiple times, or use insert_many() once, which is faster. To do so, just create a list of documents (in this case accounts holding some data) and insert that list as an argument to insert_many(), for example:
acc_1 = { "account": "stoodkev"}
acc_2 = { "account": "fabiyamada", "account_id": 261379}
acc_3 = {
"account": "jedigeiss",
"slogan": "IT Nerd, Risk Specialist, Musician, Cryptocoin Enthusiast, Banker, Gamer"
}
acc_list = [acc_1, acc_2, acc_3]
result = coll.insert_many(acc_list)
print('ObjectIds: {}'.format(result.inserted_ids))
ObjectIds: [ObjectId('5ae46693dd58330cd6660570'), ObjectId('5ae46693dd58330cd6660571'), ObjectId('5ae46693dd58330cd6660572')]
Nota bene 1: Because we've now inserted multiple documents at once, I called the attribute
result.inserted_ids instead of result.inserted_id!
Nota bene 2: I deliberately chose to insert multiple accounts, all containing an "account" (name) field, but account stoodkev I gave just that one field account where fabiyamada was the only account having an accout_id field and value. MongoDB is that flexible; you just add data to a document whenever you like, we can update the other account documents to have the same fields & values later if we like, or leave them as they are.
find_one()Now that we have saved some data in the accouts collection within our test_mongo database, let's see how to find and retrieve one document given a (simple) database query. We're going to use the find_one() method to do that:
scipio = coll.find_one({"account": "scipio"})
print(scipio)
{'_id': ObjectId('5ae46285dd58330cd666056f'), 'account': 'scipio', 'slogan': "Does it matter who's right, or who's left?"}
Alright, that's exactly what we were expecting: as you can see, as an argument to the find_one() method, I passed in a key:value pair {"account": "scipio"} / dictionary, and as a result exactly one document - being the first one, and in case case only one, - is found and returned from the collection.
However, in case I would not pass any argument to the find_one() method, since scipio is the first (account) document we inserted, the same one document will be returned:
some_account = coll.find_one()
print(some_account)
{'_id': ObjectId('5ae46285dd58330cd666056f'), 'account': 'scipio', 'slogan': "Does it matter who's right, or who's left?"}
find()In case we're interested to find and retrieve multiple documents that match the given query we can use the find() method. Please note that the returned value (in case it matches at least one document) isn't (as you might have expected?) list of documents (dictionaries) but a so-called iterable Cursor object, which (at first sight) looks a little weird to work with, but has many helper methods that come in very handy (if you'll of course continue to follow along with my forthcoming Learn Python Series episodes!).
Let's just see how this works by using find() without passing in key:value pairs, meaning our intention is to retrieve the entire current accounts collection as the returned data:
all_accounts = coll.find()
print(type(all_accounts), all_accounts)
<class 'pymongo.cursor.Cursor'> <pymongo.cursor.Cursor object at 0x10bb37f60>
for each_account in all_accounts:
print(each_account)
{'_id': ObjectId('5ae46285dd58330cd666056f'), 'account': 'scipio', 'slogan': "Does it matter who's right, or who's left?"}
{'_id': ObjectId('5ae46693dd58330cd6660570'), 'account': 'stoodkev'}
{'_id': ObjectId('5ae46693dd58330cd6660571'), 'account': 'fabiyamada', 'account_id': 261379}
{'_id': ObjectId('5ae46693dd58330cd6660572'), 'account': 'jedigeiss', 'slogan': 'IT Nerd, Risk Specialist, Musician, Cryptocoin Enthusiast, Banker, Gamer'}
find() query using count()Just as the Python len() function (to be used for example to count the amount of elements in a list), PyMongo provides us with a method count() to return the number of matching documents for a given query.
In case we want to return (or in this case: print) all accounts (documents) stores in the accounts collection, we just first run the find() method (without a key:value pair query as an argument) on our accounts collection and then append the count() method to it in order to the return the number of documents found. Like so:
num_accounts = coll.find().count()
print(num_accounts)
4
In this episode, I talked about MongoDB, being a type of NoSQL database environment, specifically a "document datastore", as opposed to SQL / relational databases. We discussed the differences between SQL vs NoSQL, and I hopefully got you to be just as enthusiastic as I am myself about using MongoDB together with Python.
We then briefly talked about how to install and run MongoDB, PyMongo as its Python driver, and Robo 3T / RoboMongo to conveniently view (and / or interact with) your MongoDB databases.
We then discussed databases, collections and documents, how to insert one or more documents, how to retrieve / find one or more documents, and how to count the number of documents in a collection matching a certain query (even a blank one).
But there's much more we can do with MongoDB and PyMongo & Python! We barely scratched the surface of all possibilities, so let's expand on what we learned today in the next sub-parts regarding PyMongo! See you in the next episode!