You know that linear regression is a popular technique and you might as well seen the mathematical equation of linear regression. But do you know how to implement a linear regression in Python?? If so don’t read this post because this post is all about implementing linear regression in Python. There are several ways in which you can do that, you can do linear regression using numpy, scipy, stats model and sckit learn. But in this post I am going to use scikit learn to perform linear regression.
Scikit-learn is a powerful Python module for machine learning. It contains function for regression, classification, clustering, model selection and dimensionality reduction. Today, I will explore the sklearn.linear_model module which contains “methods intended for regression in which the target value is expected to be a linear combination of the input variables”.
In this post, I will use Boston Housing data set, the data set contains information about the housing values in suburbs of Boston. This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository. UCI machine learning repository contains many interesting data sets, I encourage you to go through it.
So come on lets have fun with linear regression,
Exploring Boston Housing Data Set
The first step is to import the required Python libraries into Ipython Notebook.
This data set is available in sklearn Python module, so I will access it using scikitlearn. I am going to import Boston data set into Ipython notebook and store it in a variable called boston.
The object boston is a dictionary, so you can explore the keys of this dictionary.
I am going to print the feature names of boston data set.
I will see the description of this data set to know more about it. In this data set I have 506 instances(rows) and 13 attributes or parameters(columns). The goal of this exercise is to predict the housing prices in boston region using the features given.
I am going to convert boston.data into a pandas data frame.
As you can see the column names are just numbers, so I am going to replace those numbers with the feature names.
boston.target contains the housing prices.
I am going to add these target prices to the bos data frame.
Scikit Learn
In this section I am going to fit a linear regression model and predict the Boston housing prices. I will use the least squares method as the way to estimate the coefficients.
Y = boston housing price(also called “target” data in Python)
and
X = all the other features (or independent variables)
First, I am going to import linear regression from sci-kit learn module. Then I am going to drop the price column as I want only the parameters as my X values. I am going to store linear regression object in a variable called lm.
If you want to look inside the linear regression object, you can do so by typing LinearRegression. and the press <tab> key. This will give a list of functions available inside linear regression object.
Important functions to keep in mind while fitting a linear regression model are:
lm.fit() -> fits a linear model
lm.predict() -> Predict Y using the linear model with estimated coefficients
lm.score() -> Returns the coefficient of determination (R^2). A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model.
You can also explore the functions inside lm object by pressing lm.<tab>
.coef_ gives the coefficients and .intercept_ gives the estimated intercepts.
Fitting a Linear Model
I am going to use all 13 parameters to fit a linear regression model. Two other parameters that you can pass to linear regression object are fit_intercept and normalize.
In [20]: lm.fit(X, bos.PRICE)
Out[20]: LinearRegression(copy_X=True, fit_intercept=True, normalize=False)
I am going to print the intercept and number of coefficients.
I then construct a data frame that contains features and estimated coefficients.
As you can see from the data frame that there is a high correlation between RM and prices. Lets plot a scatter plot between True housing prices and True RM.
As you can see that there is a positive correlation between RM and housing prices.
Predicting Prices
I am going to calculate the predicted prices (Y^i) using lm.predict. Then I display the first 5 housing prices. These are my predicted housing prices.
Then I plot a scatter plot to compare true prices and the predicted prices.
You can notice that there is some error in the prediction as the housing prices increase.
Lets calculate the mean squared error.
The mean squared error has increased. So this shows that a single feature is not a good predictor of housing prices.
Training and validation data sets
In practice you wont implement linear regression on the entire data set, you will have to split the data sets into training and test data sets. So that you train your model on training data and see how well it performed on test data.
How not to do train-test split:
You can create training and test data sets manually, but this is not the right way to do, because you may be training your model on less expensive houses and testing on expensive houses.
How to do train-test split:
You have to divide your data sets randomly. Scikit learn provides a function called train_test_split to do this.
I am going to build a linear regression model using my train-test data sets.
Then I calculate the mean squared error for training and test data.
Input:
print “Fit a model X_train, and calculate MSE with Y_train:”, np.mean((Y_train – lm.predict(X_train)) ** 2)
print “Fit a model X_train, and calculate MSE with X_test, Y_test:”, np.mean((Y_test – lm.predict(X_test)) ** 2)
Output:
Fit a model X_train, and calculate MSE with Y_train: 19.5467584735 Fit a model X_train, and calculate MSE with X_test, Y_test: 28.5413672756
Residual Plots
Residual plots are a good way to visualize the errors in your data. If you have done a good job then your data should be randomly scattered around line zero. If you see structure in your data, that means your model is not capturing some thing. Maye be there is a interaction between 2 variables that you are not considering, or may be you are measuring time dependent data. If you get some structure in your data, you should go back to your model and check whether you are doing a good job with your parameters.
Conclusion
To recap what I have done till now,
The post How to run Linear regression in Python scikit-Learn appeared first on Big Data Made Simple - One source. Many perspectives..
The essence of education traditionally has involved the transition of accumulated knowledge to younger generations for most of its history. However, as the working routine changes from repetitive work to the knowledge-based activity, the requirements put to the quality of education have changed. So, how are big data and the resulting algorithms influencing teaching and helping students learn?
It can analyze how they’re doing
Because the data sets of student learning are so diverse, very slight nuances about how students are doing can be teased out. In this way, Arizona State is analyzing the keystrokes of the students using their devices to measure how well they are progressing, how they are struggling and what their weak and strong points are.
This, in turn, means that they can step in and help students long before they themselves might even be aware they’re in trouble.
Personalized programs
For the longest time, education assumed that one size fits all. It doesn’t matter how strong or weak a student’s skill is, it is better to put them among people of their own age and let them absorb whatever is being taught in that year.
Of course, this wasn’t just down to convention. It was also related to the difficulties of tracking what every student in a school needed and creating individual programs that best suited their learning styles.
Big data is changing that. As we gather more and more information about students, we don’t only get a better idea of how they’re doing, but can adjust the syllabus to better suit their learning needs. Even better, as this is automatic, this can be done for every student without overtaxing the teachers’ capability.
Evaluation without bias
Another way in which these evaluations differ is that they are no longer witnessed through the prism of a teacher’s likes and dislikes. For the longest time, we’ve known that teachers favor some students over others. For example, teachers tend to give higher grades to more attractive students. This does not happen consciously and instead is the result of how we’re put together, which makes it an incredibly difficult problem to tackle.
Big data offers a way out. After all, a computer does not recognize a student by their race, sex, or visual appeal. Similarly, big data can consider a test in absolute isolation – not giving the benefit of the doubt to students that have done better on previous exams. This creates an equal playing field where we are judged based on how we’re performing instead of all the external factors that surround it but should be irrelevant.
It can boost engagement
By exploring the numbers produced by 100s of thousands of students working on software, it will become far easier to know what is interesting to students and what is not.
Big data will then give a possibility of customization the learning experience to make what students are learning directly relevant to them. Even better, the technology will be able to analyze future lessons and use what it has learned from the student in previous encounters to modify the material as well as predict how hard the student will find it and how much time they will need.
It will even be able to conclude when students should take breaks and when they’re best served to study alone or in a group.
Fitting the right personalities together
Big data will make group projects as productive as possible. Right now, students are often grouped based on where they’re sitting in the class or who are their friends. The thing is, though likability is certainly a useful factor in deciding who to work with, it is certainly not the only way to do so.
A much better idea is to find which students are going to be the most useful to each other and group based on that. This will avoid the popularity contest whereby students that everybody wants to work with have too many choices while less popular students have too few.
Similarly, because students are grouped based on who they will work best with, class engagement will rise and struggling students can be brought back up to speed by who they work with rather than the teacher alone.
Last words
As we pay more attention to engaging all the stages of one’s memorization process, we’re going to see a revolution the learning curriculum is adopted on the fly based on a student’s energy levels, current interests and even how well they are able to focus.
Thus, having a good idea of what each student is capable of in turn will mean they can be advised in one of the most difficult decisions we all have to make, what we will do after school. In this way, big data will not just revolutionize the classroom, but even the choices we will make afterwards. That’s exciting (and perhaps a little bit scary).
The post How do algorithms influence teaching and bridge the students’ knowledge gap appeared first on Big Data Made Simple - One source. Many perspectives..
The General Data Protection Regulation, or GDPR enforcement date, is quickly closing in. It’s a sweeping and relatively new form of legislation that will change the current landscape considerably. How so? Well, it puts a lot of pressure on organizations to bolster advanced data security and privacy.
More specifically, it’s a European regulation that will go into effect on May 25, 2018. It’s an update to the existing directives or legislature called the Data Protection Directive.
What Is GDPR and What Will It Do?
The General Data Protection Regulation (GDPR) — which will be enforced across the entirety of Europe, including the UK — is meant to offer protection and privacy controls to the country’s citizens. Not only will they have more power, but a unified set of rules and standards are also being established to boost security and protections. There are a variety of new precautions and processes organizations will need to follow concerning all information, but mostly consumer data.
Although it’s strictly an EU law, that doesn’t mean it won’t spread elsewhere. In fact, the GDPR will have a global impact when it goes into effect. Any business, brand or team that holds, collects, stores or processes personal data from citizens of the EU will need to follow the guidelines and specifications outlined in the law. It doesn’t envelope consumers alone, either — it also includes your employees, personnel, clients and prospective clients based in the EU.
Should you choose to ignore the law, you can face fines of up to €20m or 4% of your global annual turnover.
What Rules Apply?
The rules set forth by the GDPR are quite complex. But, we can break them down so they’re easier to understand and follow — here’s an excellent infographic that accomplishes the same.
GDPR restrictions adhere to the following concepts:
Not much should stand out from standard security operations — that is, if you already value the privacy and protection of the data you have available to you. And you should, because just about every form of information is stored and accessible from the cloud these days, including your shopping habits, the places you visit, the conversations you have with friends and family, your emails, your medical records and too much more.
New York Times best-selling author Shawn DuBravac, says it best in his latest book, Digital Destiny: How the New Age of Data Will Transform the Way We Work, Live, and Communicate.
“We don’t keep things locked in our hard drives [anymore], instead we let services like Dropbox store them for us, just as a bank store most of our money.”
Today, everything is stored as data, most likely on a public network. Security and privacy should be a primary concern for every business and or individual.
Customers, personnel and anyone else you serve all care about their privacy and security. It’s just good business to at least enforce security protocols and make sure the data you are responsible for is, for the most part, protected. Don’t store raw data on a public server, always encrypt sensitive information and content, deploy the appropriate authentication measures and systems and maintain proper security — via audits — over time.
What Impact Will It Have on Businesses?
Compliance and adherence to regulatory measures are the number one concerns for lawmakers and would-be enforcers. That means these concepts should also be your utmost priorities when serving EU citizens if they aren’t already.
Farming out or “siloing” security is not a good idea in the current landscape — heck, it never was. Instead, you’ll need to ensure you embed security, protection, privacy and related protocols into the fabric and foundation of your business.
The most significant shift, of course, will be educating and training every single person, employee and partner involved in the data chain. There can be no more weak links, per se. Everyone needs to get involved, and everyone needs to work together to maintain proper security and privacy.
Expect for your spending on security to increase, but also for the training, deployment, tools and software to become prominent concerns — and investments.
The post What does General Data Protection Regulation (GDPR) mean for you? appeared first on Big Data Made Simple - One source. Many perspectives..
2017 was one of the worst years for data breaches yet, and it’s only downhill from here. Here’s what you need to do to prepare yourself.
These days, it seems like it’s no longer a question of if your personal information will wind up in the hands of hackers, but when. You can’t even tune into the news anymore without hearing a security failure or data breach. Fancy Bear’s attack on US Defense Contractors and attempted disruption of the Olympic games are the latest stories to make the rounds, and you can bet money that once the news cycle fades on those two, new breaches will appear to fill the void.
Sensitive data – enterprise, public sector, or otherwise – has never been targeted with quite so much frequency, and the threat surface facing modern organizations has never been greater. You need to start accounting for that. Because if you don’t, your organization might be next on the long list of breach victims.
But where exactly can you start? What can you do to protect your business and its data? How can you ensure that in this era of massive cyber attacks and digital espionage, you come out unscathed?
First, Know What You Need To Protect
The most important tool in your battle against the digital threat landscape is knowledge. You need to know what data your business needs to protect, where that data is stored, who uses that data, how they access it, and why. The most important thing is to understand the flows of data across your network.
For instance, a document containing marketing information on a product launch will likely be shared with both internal staff and external stakeholders. Both parties might open that document on a myriad selection of devices, from smartphones and tablets to laptops and desktops. Understand where that access takes place, and do your best to identify any potential vulnerabilities (an unsecured wireless network, for example).
Second, Know Your Threat Profile
What sort of criminals would want to target your organization, and why?
Are you a healthcare provider that works with data which is vulnerable to ransomware or can be sold for a mint on the black market? Are you a defense contractor that works with classified information a foreign power might want to get their hands on? Are you a consumer device manufacturer whose proprietary blueprints could destroy your competitive advantage if they’re leaked?
That’s your most valuable data, but it’s not the only information that might be compromised. Client lists, employee information, and financial data are all vulnerable as well, regardless of industry or vertical. By acknowledging this – and understanding the intent of the criminals targeting your organization – you can better prepare yourself against them.
Third, Talk To Your Employees
No matter what sort of security measures you put in place and no matter how ironclad your infrastructure, your employees will always be the weakest link. Hackers know that – it’s why phishing scams are still one of the most popular avenues of attack. It’s a lot easier to fool a tired but well-meaning staffer into clicking on a link than it is targeting an expensive, complex security system, after all.
You need mandatory security training to mitigate at least some of the risk here. Coach employees on the importance of cybersecurity, and help them understand what they’re protecting and why. It won’t prevent human-based cyberattacks altogether – everyone makes mistakes, even you – but it will allow you to mitigate the risk.
And solutions do exist to address this problem further, as well. We’ll talk more about those in a moment.
Fourth, Look At Your Partners
I recall a story I heard once about a business with a large competitive advantage over its overseas rivals, tied to a proprietary set of technologies present in all their products. This enterprise took cyber security very seriously. Its security perimeter was nigh impenetrable, and its workers all regularly had to undergo rigorous security training.
It still ended up getting breached.
See, a black hat group hired by one of the company’s rivals saw its formidable security posture, and concluded that it wouldn’t be possible to hack the company directly. Instead, it started examining the organization’s business partners. Eventually, the group noticed that one of the manufacturers the organization worked with had incredibly lax security – and that was when they struck.
Next thing the business knew, its designs were in the hands of its competitors, and its advantage in the market was gone.
The lesson in this story is simple. Even if your own security is some of the best in your industry, you cannot trust that your business partners will have the same posture as you. While working only with organizations that have proven cybersecurity can certainly help in that regard, you need to do more.
Finally, Put Your Defenses In Place
With all the necessary knowledge in place – the threats facing your business, where your sensitive data is stored, how it’s accessed and used, and your network map – you can now finally lay out your security plans. For this stage, it’s easiest to break what you need to secure down into four categories. These are people, apps, systems, and data.
For Apps, Consider The Following
For People, Consider The Following
For Systems, Consider The Following
For Data, Consider The Following
Cyber attacks and data breaches have reached an all-time high. Unfortunately, it’s going to get worse before it gets better. Only with a comprehensive approach to an understanding of cybersecurity can you protect yourself, and keep your data out of the hands of the people who would misuse it.
The post US data breaches have hit an all-time high – How to protect your business appeared first on Big Data Made Simple - One source. Many perspectives..
Employee well-being is a major concern for organizations, especially when those entities are concerned about productivity.
Data from the Bureau of Labor Statistics published a report revealing that in 2016, there were 2.9 million reported workplace-related injuries and illnesses.
Nearly one-third of them required workers to take days off work. Those are significant numbers considering that many workers do not formally make reports to their bosses when they get sick.
In the United Kingdom, data shows there were 25.7 million days of work lost in 2016 and 2017 due to illnesses. Also, 12.5 million of those cases were related to depression, anxiety or stress.
It’s not surprising that many employers are investigating ways to take proactive measures to prevent employees from getting sick.
One of their main efforts involves using big data and poring over statistics that could indicate instances of ill health are on the rise or going down. However, both potential positive and negative aspects of that approach exist. Here are some thoughts.
People Can Get Healthier Together
Many of the advantages of collecting employee data involve people teaming up to track their metrics over time and work toward a common goal, such as weight loss. In those cases, participants can encourage each other and see the changes in their colleagues.
Employers Experience Cost Savings
When employees can’t work due to illness, they frequently cause their colleagues to bear the burdens of their absence and may disrupt operations in the process, creating new expenses. Also, if key individuals working on critical projects get sick, organizations could face costly consequences due to missed deadlines.
When employees are healthier, workplace representatives may choose less expensive, more appropriate health insurance premium packages, too.
Employees Could Feel Discriminated Against
Some companies reportedly track employee health data to see how many employees are likely to become pregnant.
Women frequently already experience a great deal of anxiety about telling their employers they’ll need to take maternity leave or otherwise adjust their work schedules due to pregnancy, and this new development could make that worse.
One app called Castlight gathers data about employees and uses it to urge them to make better decisions about their health. Not surprisingly, some individuals assert that practice is too invasive.
The company says it cannot give organizations data about individual employees, but that does not always make people feel better.
That’s because current laws give more freedom to sort through health data that does not identify a person compared to the material that does.
Even if a worker’s data is represented in a larger segment, he or she might wonder if data pulled from a software suite is causing a boss to have unfavorable views of the individual’s dietary choices, decision to smoke cigarettes or the fact that he or she drinks lots of soda at work.
Fearful Feelings May Increase
Also, individuals are already wary about how the apps they use collect and evaluate data about them. That’s because many of the apps do it silently in the background.
Sometimes, the data collection practices are part of the terms of use for an application, so if users do not consent, they cannot access the app.
If people do not understand how their workplaces use collected information and feel they cannot go to a designated individual or department to ask questions, they may become so uneasy that their work outputs decrease.
If the pressure feels too great, they may look for other employment prospects.
Being Transparent Is Often Preferable
When an organization decides to start using big data for employee tracking purposes, that conclusion could mean many things.
Some specifics must be determined. For example, what statistics are gathered, and why? Also, are employees aware of the data tracking methods, and can they opt out of them without fear of being seen as non-compliant?
It’s crucial for workplaces to keep ethics in mind at all times when collecting and using data. They must determine the best ways to protect employee privacy while meeting organizational objectives.
Coming up with an information governance plan and asking for employees’ input is a great start. Furthermore, workers should get the opportunity to formally say they do not consent to their details being collected and not have to give reasons why that’s the case.
Making employees aware of what to do if they have questions about data use at work is another excellent step to take.
Also, employers should strive to show they’re genuinely open to receiving feedback. Being able to provide it should make employees feel they still have some control over data about themselves and how it is used.
Health data collection is a practice likely to continue gaining popularity in modern workplaces.
However, keeping employees on board with the idea and not making them consider working elsewhere involves honesty and openness about the techniques used and what purposes they serve.
The post The pros and cons of using big data for employee wellness appeared first on Big Data Made Simple - One source. Many perspectives..
Chatbots are conversational tools, capable of engaging multiple users to give structured responses to the most basic queries. Due to their instantaneity and ease of use, they have demonstrated significant potential on E-Commerce websites. As users face issues getting to their desired product page on the website and making a purchase decision thereafter, chatbots can prove to be useful for enticing these users that are about to leave the website without purchasing.
In addition to increasing overall efficiency of customer service, chatbots can also be successfully targeted on relevant product pages to display offers and promotions in order to persuade users that are on the page. If used effectively, they could be great learning tools to know more about your customer’s shopping habits with the intention to offer more relevant products in the future. As research shows that more than 80% of businesses will have some sort of automation through chatbots by 2020; the future for chatbots in e-commerce does look very promising indeed.
This infographic by Market Inspector shows how chatbots optimize a customer’s online purchase experience using the most recent user surveys.
The post Unlocking the potential of chatbots in eCommerce (Infographic) appeared first on Big Data Made Simple - One source. Many perspectives..
