Big data is a top trending buzzword. But, unlike overused buzzwords such as ‘omnichannel marketing’ or ‘growth hacking’, big data is very underhyped. According to IBM, 62% of retailers report that the use of big data is giving them a serious competitive advantage. Knowing what your customer wants and when they want it can be available at your fingertips with big data; all you need are the right tools and processes in place to make use of it. Let’s explore 7 innovative examples of big data personalization in retail for some inspiration.
Check out our previous article to see if you’re operating in one of the ten business areas that should be using big data already.
Macy’s: The Traditional Department Store is Ahead of Its Time
This upmarket department store has a long history of providing excellent customer service and has become a household name. Despite the heritage established since the first store opened in 1858, the brand has taken to the digital age like a fish to water.
Macy’s uses big data to offer a smarter customer experience. The brand analyzes multiple data points, such as stock levels and price promotions, and combines these findings with stock keeping unit data from a product at a particular location – as well as customer data – to ascertain which products are on sale in each store. This ensures that its chosen products suit the buying habits of customers in each location.
On top of this, Macy’s collects customer data ranging from visit frequency to style preference. This data is used to personalize the customer experience, offering incentives at the point of sale with loyalty rewards and promotions. This data also enables it to send targeted direct mail to its customers to boost conversions.
Amazon’s purchase recommendation engine
The ecommerce heavyweight Amazon has truly mastered its recommendation engine, but its functionality is actually quite simple. The algorithm is based on a user’s purchase history, the items they have in their cart already, items they have rated or liked in the past, and what other customers have viewed or purchased recently. In fact, it has been reported that over 35% of all Amazon sales are generated by the recommendation engine – a testament to the importance of product recommendations.
The primary reason for recommendation engine is to address the ‘long-tail problem’ – the fact that rare or obscure items are frequently not searched for, and therefore don’t drive revenue. By recommending long-tail items to shoppers, you can seriously drive the ROI potential of slower-moving ecommerce listings.
Kohl’s
Kohl’s is a brand with big data plans. This brand has recently suffered a decline in sales of 2.4%, along with decreased shopper traffic, and the brand’s CEO contemplated closing around 1,100 stores. However, in a change of heart, the brand has decided to implement new technologies to streamline its shopping experience and make stores smaller. To achieve this, it has invested over $2 billion in tech and big data initiatives. Product recommendations aside, the brand is on a mission to use big data firstly for the benefit of its customers, as well as to make the stores more profitable.
The entire online and physical shopping experience is personalized, from when a visitor lands on the homepage and is faced with deals and products on every page, to personalized offers that counter shopping cart abandonment. Kohl’s also uses its big data to create tailored marketing campaigns, which have been produced with customer data in mind. The brand now plans for data science to assist merchandising allocation, including external data like macro-economic conditions and social data, which will determine which products are stocked. This will ensure that products fly off the shelves faster.
Mall of America navigator chatbots
IBM has provided the Mall of America with a chatbot named E.L.F to assist shoppers navigating the vast complex. The Mall of America is in Bloomington, Minnesota, and it is the largest shopping complex in the northern states. It plays host to 520 retailers, 50 restaurants, 14 movie theaters, 2 hotels, an indoor theme park and a museum.
E.L.F. can create personalized shopping itineraries for each customer, finding the right experience for them (dependent on their needs). The chatbot is operated by a simple interface akin to a text messaging platform. E.L.F. is available via the Facebook Messenger app, the browser page, or kiosks in the Mall of America.
Nordstrom: fusing the online and offline shopping experience
This luxury retailer has mastered harnessing big data to fuse online and offline shopping experiences. Nordstom’s marketing team tracks Pinterest pins in order to identify which products are trending, and then employs this data to promote the right products in its physical stores.
Over 30% of Nordstrom’s budget is spent on technology, having established the ‘Nordstrom Innovation Lab’ based in Seattle for product development and testing. On top of this, Nordstrom hosts interactive touchscreens in changing rooms to allow customers to order products and view stock online.
TopShop
TopShop has been experimenting with new technologies to implement augmented reality into its shopping experience since 2010. Flagship stores have virtual fitting rooms where customers can select clothes to see how they would look wearing them on a screen. This saves the customer the time and effort of trying on clothes themselves.
In 2015, TopShop partnered with Twitter to analyze real time data on the social network, and identified trends as they happened during the five day London Fashion week event. These trends were grouped together on billboards using Twitter hashtags, so customers walking by would be encouraged to tweet a hashtag to their TopShop account indicating their favorite products. The fashion retailer then responded with a curated collection of the top picks.
This novel use of big data ensured that TopShop knew exactly what its customers were looking to buy following London Fashion Week.
IKEA
The Swedish interior giant IKEA featured image recognition and augmented reality for the first time when it showcased its 2013 catalog. Customers could scan through the catalog with their mobile devices to highlight products they were interested in, and from this, the brand offered personalized digital content and reviews to inform their purchase. The brand also used image-recognition technology, with which customers can scan catalog items and virtually place them in their own homes to see what they would look like. They can then select the colors and sizes that work best in the space, without having to actually go to store and purchase the product. This allowed the catalog readers to make informed purchases, resulting in higher customer satisfaction and fewer returned items.
These innovative uses of big data really enhance the customer experience, and have the potential to boost your sales. You don’t have to be a big player in retail to use big data. You could use it yourself to get ahead of your competitors, particularly if you use a Shopify storefront. This platform integrates with Blendo, a big data analytics plugin. Plugins and apps can be very useful ways for you to automatically collect and pull up data from multiple sources to inform your business decisions.
How will you integrate big data into your retail business? Have any of these examples inspired you? Let us know in the comments.
The post 7 examples of big data retail personalization appeared first on Big Data Made Simple - One source. Many perspectives..
So you have decided to learn Python, but you don’t have prior programming experience. So you are confused on where to start, and how much Python to learn. These are some of the common questions a beginner has while getting started with Python(for data centric application).
“How long does it take to learn Python”
“How much Python should I learn for performing data analysis”
“What are the best books/courses to learn Python”
“Should I be an expert Python programmer, in order to work with data sets”
It is good to be confused, while beginning to learn a new skill, that’s what author of “learn anything in 20 hours” says. Don’t Panic, I will show you how to get started quickly without becoming a coding ninja in Python.
Don’t make the mistake I did
Before getting started with Python, I had a misconception that for performing data analysis in Python, I have to be proficient in Python programming. So I took Udacity’s intro Python programming course, completed code academy Python tutorials and read several Python programming books. For 3 months(spending 3 hours per day), I was learning Python programming by completing small software projects. Coding was fun, but my goal was not to become a Python developer, but to do data analysis using Python. Then I realized that I was spending more time learning how to develop software in Python, rather than doing data analysis.
After a few hours of research, I found out that I need to learn 5 Python libraries to effectively solve a broad set of data analysis problems. Then I started learning these libraries one after the other.
In my opinion, it is not necessary to become proficient at building good software in Python to be able to productively perform data analysis.
Ignore the resources intended for general audience
While there are many excellent Python books and online courses, I wouldn’t recommend some of them as they are intended for a general audience rather than for some one who wants to do data analysis. Also there are couple of books on “Scientific Programming in Python”, but they are geared toward various topics that are mathematically-oriented rather than being about data analysis and statistics. Don’t waste your time, by taking courses and reading books that are intended for general audience.
Before proceeding further, first set up your programming environment, and learn how to work in IPython notebook.
Learning Pathway
Start with code academy, complete all the exercises in code academy. You can complete the exercises in 20 days, by investing 3 hours per day. Code academy covers all the basic Python concepts. But it doesn’t follow a project oriented approach like Udacity; that doesn’t matter, because your goal is to work on data science projects, not on building software using Python.
After completing the code academy exercises go through this I python notebook:
Python Essentials Tutorials (I have provided the links to download the file in conclusion part)
It consists of concepts that are not covered in code academy.You can complete this tutorial within an hour or two.
Now you know enough basics to start learning Python libraries.
Numpy
First, start learning NumPy as it is the fundamental package for scientific computing with Python. A good understanding of Numpy will help you use tools like Pandas effectively.
I have prepared an IPython note book, that includes the basic concepts of Numpy. The tutorial covers the most frequently performed operations in Numpy, such as, working with N-dimensional array, Indexing and slicing of arrays, Indexing using integer arrays, transposing an array, universal functions, data processing using arrays, frequently used statistical methods, etc.
Pandas
Pandas contain high level data structures and manipulation tools to make data analysis fast and easy in Python.
Tutorial includes working with series, data frames, dropping entries from an axis, working with missing values, etc.
Matplotlib
This is a four part Matplotlib tutorial.
1st part:
First part introduces the basic functionalities of Matplotlib, the basic figure types.
2nd part:
Covers how to control the style and color of a figure, such as markers, line thickness, line patterns and using color maps.
3rd part:
Annotation of a figure- compositing several figures, controlling the axis range, aspect ratio and coordinate system.
4th part:
Covers working with complex figures.
Conclusion
One of the easiest mistakes you can make when learning Python is attempting to learn too many libraries at the same time. When you try to learn everything at once, you spend too much time switching between different concepts, getting frustrated, and move on to something else.
So focus stick on to this process:
Understand Python basics
Learn Numpy
Learn Pandas
Learn Matplotlib
Links to Download:
You can download the files from my github account. The files are in .ipynb format. The files also includes the pictures I have used for illustration.
3) Matplotlib
If you have any queries, feel free to ask in comments.
The post Step by step approach to perform data analysis using Python appeared first on Big Data Made Simple - One source. Many perspectives..
MySQL is the second most widely used open-source relational database management system in the world. It has become so popular because of its consistent fast performance, high reliability and ease of use. This article presents some of the best practices in MySQL.
1. Always use proper datatype
Use datatypes based on the nature of data. If you use irrelevant datatypes it may consume more space or may lead to errors.
Example: Using varchar (20) to store date time values instead of DATETIME datatype will lead to errors during date time-related calculations and there is also a possible case of storing invalid data.
2. Use CHAR (1) over VARCHAR(1)
If you string a single character, use CHAR(1) instead of VARCHAR(1) because VARCHAR(1) will take extra byte to store information
3. Use CHAR datatype to store only fixed length data
Example: Using char(1000) instead of varchar(1000) will consume more space if the length of data is less than 1000
4. Avoid using regional date formats
When you use DATETIME or DATE datatype always use YYYY-MM-DD date format or ISO date format that suits your SQL Engine. Other regional formats like DD-MM-YYY, MM-DD-YYYY will not be stored properly.
5. Index key columns
Make sure to index the columns which are used in JOIN clauses so that the query returns the result fast.
If you use UPDATE statement that involves more than one table make sure that all the columns which are used to join the tables are indexed
6. Do not use functions over indexed columns
Using functions over indexed columns defeats the purpose of the index. Suppose you want to get data where first two character of customer code is AK, do not write
SELECT columns FROM table WHERE left (customer_code,2)=’AK’
but rewrite it using
SELECT columns FROM table WHERE customer_code like ‘AK%’
which will make use of index which results in faster response time.
7. Use SELECT * only if needed
Do not just blindly use SELECT * in the code. If there are many columns in the table, all will get returned which will slow down the response time particularly if you send the result to a front-end application.
Explicitly type out the column names which are actually needed.
8. Use ORDER BY Clause only if needed
If you want to show the result in front-end application, let it ORDER the result set. Doing this in SQL may slow down the response time in the multi-user environment.
9. Choose proper Database Engine
If you develop an application that reads data more often than writing (ex: search engine), choose MyISAM storage engine.
If you develop an application that writes data more often than reading (ex: real-time bank transactions), choose INNODB storage engine.
Choosing wrong storage engine will affect the performance
10. Use EXISTS clause wherever needed
If you want to check the existence of data, do not use
If (SELECT count(*) from Table WHERE col=’some value’)>0
instead, use EXISTS clause
If EXISTS(SELECT * from Table WHERE col=’some value’)
which is faster in response time.
The post Top 10 best practices in MySQL appeared first on Big Data Made Simple - One source. Many perspectives..
You know that linear regression is a popular technique and you might as well seen the mathematical equation of linear regression. But do you know how to implement a linear regression in Python?? If so don’t read this post because this post is all about implementing linear regression in Python. There are several ways in which you can do that, you can do linear regression using numpy, scipy, stats model and sckit learn. But in this post I am going to use scikit learn to perform linear regression.
Scikit-learn is a powerful Python module for machine learning. It contains function for regression, classification, clustering, model selection and dimensionality reduction. Today, I will explore the sklearn.linear_model module which contains “methods intended for regression in which the target value is expected to be a linear combination of the input variables”.
In this post, I will use Boston Housing data set, the data set contains information about the housing values in suburbs of Boston. This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository. UCI machine learning repository contains many interesting data sets, I encourage you to go through it.
So come on lets have fun with linear regression,
Exploring Boston Housing Data Set
The first step is to import the required Python libraries into Ipython Notebook.
This data set is available in sklearn Python module, so I will access it using scikitlearn. I am going to import Boston data set into Ipython notebook and store it in a variable called boston.
The object boston is a dictionary, so you can explore the keys of this dictionary.
I am going to print the feature names of boston data set.
I will see the description of this data set to know more about it. In this data set I have 506 instances(rows) and 13 attributes or parameters(columns). The goal of this exercise is to predict the housing prices in boston region using the features given.
I am going to convert boston.data into a pandas data frame.
As you can see the column names are just numbers, so I am going to replace those numbers with the feature names.
boston.target contains the housing prices.
I am going to add these target prices to the bos data frame.
Scikit Learn
In this section I am going to fit a linear regression model and predict the Boston housing prices. I will use the least squares method as the way to estimate the coefficients.
Y = boston housing price(also called “target” data in Python)
and
X = all the other features (or independent variables)
First, I am going to import linear regression from sci-kit learn module. Then I am going to drop the price column as I want only the parameters as my X values. I am going to store linear regression object in a variable called lm.
If you want to look inside the linear regression object, you can do so by typing LinearRegression. and the press <tab> key. This will give a list of functions available inside linear regression object.
Important functions to keep in mind while fitting a linear regression model are:
lm.fit() -> fits a linear model
lm.predict() -> Predict Y using the linear model with estimated coefficients
lm.score() -> Returns the coefficient of determination (R^2). A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model.
You can also explore the functions inside lm object by pressing lm.<tab>
.coef_ gives the coefficients and .intercept_ gives the estimated intercepts.
Fitting a Linear Model
I am going to use all 13 parameters to fit a linear regression model. Two other parameters that you can pass to linear regression object are fit_intercept and normalize.
In [20]: lm.fit(X, bos.PRICE)
Out[20]: LinearRegression(copy_X=True, fit_intercept=True, normalize=False)
I am going to print the intercept and number of coefficients.
I then construct a data frame that contains features and estimated coefficients.
As you can see from the data frame that there is a high correlation between RM and prices. Lets plot a scatter plot between True housing prices and True RM.
As you can see that there is a positive correlation between RM and housing prices.
Predicting Prices
I am going to calculate the predicted prices (Y^i) using lm.predict. Then I display the first 5 housing prices. These are my predicted housing prices.
Then I plot a scatter plot to compare true prices and the predicted prices.
You can notice that there is some error in the prediction as the housing prices increase.
Lets calculate the mean squared error.
The mean squared error has increased. So this shows that a single feature is not a good predictor of housing prices.
Training and validation data sets
In practice you wont implement linear regression on the entire data set, you will have to split the data sets into training and test data sets. So that you train your model on training data and see how well it performed on test data.
How not to do train-test split:
You can create training and test data sets manually, but this is not the right way to do, because you may be training your model on less expensive houses and testing on expensive houses.
How to do train-test split:
You have to divide your data sets randomly. Scikit learn provides a function called train_test_split to do this.
I am going to build a linear regression model using my train-test data sets.
Then I calculate the mean squared error for training and test data.
Input:
print “Fit a model X_train, and calculate MSE with Y_train:”, np.mean((Y_train – lm.predict(X_train)) ** 2)
print “Fit a model X_train, and calculate MSE with X_test, Y_test:”, np.mean((Y_test – lm.predict(X_test)) ** 2)
Output:
Fit a model X_train, and calculate MSE with Y_train: 19.5467584735 Fit a model X_train, and calculate MSE with X_test, Y_test: 28.5413672756
Residual Plots
Residual plots are a good way to visualize the errors in your data. If you have done a good job then your data should be randomly scattered around line zero. If you see structure in your data, that means your model is not capturing some thing. Maye be there is a interaction between 2 variables that you are not considering, or may be you are measuring time dependent data. If you get some structure in your data, you should go back to your model and check whether you are doing a good job with your parameters.
Conclusion
To recap what I have done till now,
The post How to run Linear regression in Python scikit-Learn appeared first on Big Data Made Simple - One source. Many perspectives..
The essence of education traditionally has involved the transition of accumulated knowledge to younger generations for most of its history. However, as the working routine changes from repetitive work to the knowledge-based activity, the requirements put to the quality of education have changed. So, how are big data and the resulting algorithms influencing teaching and helping students learn?
It can analyze how they’re doing
Because the data sets of student learning are so diverse, very slight nuances about how students are doing can be teased out. In this way, Arizona State is analyzing the keystrokes of the students using their devices to measure how well they are progressing, how they are struggling and what their weak and strong points are.
This, in turn, means that they can step in and help students long before they themselves might even be aware they’re in trouble.
Personalized programs
For the longest time, education assumed that one size fits all. It doesn’t matter how strong or weak a student’s skill is, it is better to put them among people of their own age and let them absorb whatever is being taught in that year.
Of course, this wasn’t just down to convention. It was also related to the difficulties of tracking what every student in a school needed and creating individual programs that best suited their learning styles.
Big data is changing that. As we gather more and more information about students, we don’t only get a better idea of how they’re doing, but can adjust the syllabus to better suit their learning needs. Even better, as this is automatic, this can be done for every student without overtaxing the teachers’ capability.
Evaluation without bias
Another way in which these evaluations differ is that they are no longer witnessed through the prism of a teacher’s likes and dislikes. For the longest time, we’ve known that teachers favor some students over others. For example, teachers tend to give higher grades to more attractive students. This does not happen consciously and instead is the result of how we’re put together, which makes it an incredibly difficult problem to tackle.
Big data offers a way out. After all, a computer does not recognize a student by their race, sex, or visual appeal. Similarly, big data can consider a test in absolute isolation – not giving the benefit of the doubt to students that have done better on previous exams. This creates an equal playing field where we are judged based on how we’re performing instead of all the external factors that surround it but should be irrelevant.
It can boost engagement
By exploring the numbers produced by 100s of thousands of students working on software, it will become far easier to know what is interesting to students and what is not.
Big data will then give a possibility of customization the learning experience to make what students are learning directly relevant to them. Even better, the technology will be able to analyze future lessons and use what it has learned from the student in previous encounters to modify the material as well as predict how hard the student will find it and how much time they will need.
It will even be able to conclude when students should take breaks and when they’re best served to study alone or in a group.
Fitting the right personalities together
Big data will make group projects as productive as possible. Right now, students are often grouped based on where they’re sitting in the class or who are their friends. The thing is, though likability is certainly a useful factor in deciding who to work with, it is certainly not the only way to do so.
A much better idea is to find which students are going to be the most useful to each other and group based on that. This will avoid the popularity contest whereby students that everybody wants to work with have too many choices while less popular students have too few.
Similarly, because students are grouped based on who they will work best with, class engagement will rise and struggling students can be brought back up to speed by who they work with rather than the teacher alone.
Last words
As we pay more attention to engaging all the stages of one’s memorization process, we’re going to see a revolution the learning curriculum is adopted on the fly based on a student’s energy levels, current interests and even how well they are able to focus.
Thus, having a good idea of what each student is capable of in turn will mean they can be advised in one of the most difficult decisions we all have to make, what we will do after school. In this way, big data will not just revolutionize the classroom, but even the choices we will make afterwards. That’s exciting (and perhaps a little bit scary).
The post How do algorithms influence teaching and bridge the students’ knowledge gap appeared first on Big Data Made Simple - One source. Many perspectives..
The General Data Protection Regulation, or GDPR enforcement date, is quickly closing in. It’s a sweeping and relatively new form of legislation that will change the current landscape considerably. How so? Well, it puts a lot of pressure on organizations to bolster advanced data security and privacy.
More specifically, it’s a European regulation that will go into effect on May 25, 2018. It’s an update to the existing directives or legislature called the Data Protection Directive.
What Is GDPR and What Will It Do?
The General Data Protection Regulation (GDPR) — which will be enforced across the entirety of Europe, including the UK — is meant to offer protection and privacy controls to the country’s citizens. Not only will they have more power, but a unified set of rules and standards are also being established to boost security and protections. There are a variety of new precautions and processes organizations will need to follow concerning all information, but mostly consumer data.
Although it’s strictly an EU law, that doesn’t mean it won’t spread elsewhere. In fact, the GDPR will have a global impact when it goes into effect. Any business, brand or team that holds, collects, stores or processes personal data from citizens of the EU will need to follow the guidelines and specifications outlined in the law. It doesn’t envelope consumers alone, either — it also includes your employees, personnel, clients and prospective clients based in the EU.
Should you choose to ignore the law, you can face fines of up to €20m or 4% of your global annual turnover.
What Rules Apply?
The rules set forth by the GDPR are quite complex. But, we can break them down so they’re easier to understand and follow — here’s an excellent infographic that accomplishes the same.
GDPR restrictions adhere to the following concepts:
Not much should stand out from standard security operations — that is, if you already value the privacy and protection of the data you have available to you. And you should, because just about every form of information is stored and accessible from the cloud these days, including your shopping habits, the places you visit, the conversations you have with friends and family, your emails, your medical records and too much more.
New York Times best-selling author Shawn DuBravac, says it best in his latest book, Digital Destiny: How the New Age of Data Will Transform the Way We Work, Live, and Communicate.
“We don’t keep things locked in our hard drives [anymore], instead we let services like Dropbox store them for us, just as a bank store most of our money.”
Today, everything is stored as data, most likely on a public network. Security and privacy should be a primary concern for every business and or individual.
Customers, personnel and anyone else you serve all care about their privacy and security. It’s just good business to at least enforce security protocols and make sure the data you are responsible for is, for the most part, protected. Don’t store raw data on a public server, always encrypt sensitive information and content, deploy the appropriate authentication measures and systems and maintain proper security — via audits — over time.
What Impact Will It Have on Businesses?
Compliance and adherence to regulatory measures are the number one concerns for lawmakers and would-be enforcers. That means these concepts should also be your utmost priorities when serving EU citizens if they aren’t already.
Farming out or “siloing” security is not a good idea in the current landscape — heck, it never was. Instead, you’ll need to ensure you embed security, protection, privacy and related protocols into the fabric and foundation of your business.
The most significant shift, of course, will be educating and training every single person, employee and partner involved in the data chain. There can be no more weak links, per se. Everyone needs to get involved, and everyone needs to work together to maintain proper security and privacy.
Expect for your spending on security to increase, but also for the training, deployment, tools and software to become prominent concerns — and investments.
The post What does General Data Protection Regulation (GDPR) mean for you? appeared first on Big Data Made Simple - One source. Many perspectives..
