The word “Big data” prevailed in 2017, and it’s going to keep prevailing in the following years. In our previous post, I’ve introduced some concepts about big data, machine learning, and data mining (see post: Understanding Big data, Data mining, and Machine Learning in 5 Minutes). Now let’s dig deeper into Machine Learning with a brief walk-through of some most commonly used ML algorithms, no codes, no abstract theories, just pictures and some examples of how they are used.
The list of algorithms covered in this article include:
1. Decision Tree
Classify a set of data into different groups using certain attributes, execute a test at each node, through brach judgement, further split the data into two distinct groups, so on and so forth. Tests are done based on existing data, and when new data are being added it can be classified to the corresponding group
Classify data according to some features, whenever the process goes to the next step, there is a judging branch, and the judgement divides the data into two, and the process goes on. When tests are done with existing data, new data can be These questions are learned by the existing data, when there is new data coming in, computer can categorize data into the right leaves.
2. Random Forest
Select randomly from the original data, and form into different subsets.
Matrix S is the original data, and it contains 1-N data rows, while A, B, C are the features, and the last C stands for categories.
Create random subsets from S, let’s say we got M sets of subsets.
And we get M sets of decision trees from these subsets:
Throw new data into these trees, we can get M sets of results, and we count to see which results are the most in all M sets, we can consider that as the final result.
3. Logistic Regression
When the probability of the predicting target is larger than 0, and less than or equal to 1, it cannot be fulfilled by simple linear model. Because when domain of definition is not within certain level, the range would exceed the specified interval.
We better go with model with this kind.
So how can we get this model?
This model needs to fulfill two conditions, “Larger than or equal to 0”, “Less than or equal to 1”
And we transform the formula, we can get the logistic regressions model:
By calculating the original data, we can get corresponding coefficients.
And we get the logistic model plot.
4. Support Vector Machine
To separate the two classes from hyperplane, the best choice will be the hyperplane that leaves the maximum margin from both classes. Because Z2>Z1, so the green one is better.
Use a linear equation to express the hyperplane, class above the line is larger than or equal to 1, the other class is less than or equal to -1.
Calculate the distance between the point to the surface by using the equation in the graph:
So we get the expression of total margin as below, the aim is to maximize the margin, which we need to do is to minimize the denominator.
For example, we use 3 points to find the optimal hyperplane, define weight vector=(2, 3) – (1, 1)
And get weight vector (a, 2a), substitute these two points into the equation
When a is confirmed, the result using (a, 2a) is support vector,
Equation substituting in a and w0 is support vector machine.
5. Naive Bayes
Here’s an example of NLP:
Giving out a pieces of text, examine the text’s attitude is positive or negative.
To solve the problem, we can only look at some of the words:
And these words, will represent by only some of words and their counts.
And the original question is: Give you a sentence, which category does it belong?
By using Bayes Rules, it is going to be an easy question.
The question becomes, in this class, what’s the probability of occurrence of this sentence? And remember not to forget the other two probabilities in the equation.
Example: the probability of occurrence of the word “love” is 0.1 in the positive class, and 0.001 in the negative class.
6. k-NearestNeighbor
When comes a new datum, which category has the most points nearest to it, it belongs to which category.
For example: To distinguish “dog” and “cat”, we judge from two features, “claws” and “sound”. Circles and triangles are the known categories, what about “star”:
When K=3, these three lines connect the nearest 3 points, and circles are more, so “star” belongs to “cat”.
7. k-means
Separate the data into 3 classes, the pink part is the biggest, while the yellow is the smallest.
Pick 3, 2, 1 as default, and calculate the distance between the rest data and the defaults, and classify it into the class that has the shortest distance.
After classification, calculate the means of each class, and set it as the new center.
After some rounds, we can stop when the class no longer changes.
8. Adaboost
Adaboost is one measure of boosting.
Boosting is to gather up the classifiers that didn’t have satisfied results, and generate a classifier that may have better effect.
As the below shows, tree 1 and tree 2 don’t have good effects individually, but if we input the same data, and sum up the results, the final result will be more convincing.
An example for adaboost, in handwriting recognition, the panel can extract many features, such as the beginning direction, distance between beginning point and ending point, and etc.
When training the machine, it will get the weight of each feature, like 2 and 3, the beginnings of writing them are very similar, so this feature does little to classification, so its weight is little.
But this alpha angle has a great recognizability, so the weight of this feature will be great. The final outcome will be a result of considering all of these features.
9. Neural Network
In NN, an input may end up into at least two classes.
Neural network is formed of neures, and connections of neures.
The first layer is the input layer, and the last layer is the output layer.
In hidden layers and output layer, they both have their own classifiers.
When an input comes in the network, and being activated, the calculated score will be passed down to the next layer. Scores shown in the output layer are the scores for each class. Example below gets the result of class 1;
same input being passed to different knots generates different scores, which is because that in each knot, it has different weights and bias, and this is propagation.
10. Markov
Markov Chain consists of states and transitions.
For example, get a Markov Chain based on “the quick brown fox jumps over the lazy dog”.
First, we need to set every word under a state, and we need to calculate the probability of state transitions.
These are the probabilities calculated by one single sentence. When you use massive data of texts to train the computer, you will get a bigger state transition matrix, such as words that can follow “the”, and their corresponding probabilities.
The post 10 machine learning algorithms you should know in 2018 appeared first on Big Data Made Simple - One source. Many perspectives..
Big Data and cloud computing are now firmly established concepts in the IT world. Businesses of all sizes understand that Big Data delivers insights that were previously difficult to get from the large stores of complex data that they collect each day. Organizations know that the cloud provides scalability, cost-efficiency, and flexibility.
However, what is equally important in a Big Data and cloud computing landscape is that security does not remain an afterthought. Consider for a moment the following security breaches, each of which took place in public or private cloud computing systems:
The incidents highlighted above all arose from a similar trend—companies being in such a rush to implement cloud computing and Big Data solutions that they took a careless approach to security. When security becomes an afterthought, the potential for disaster increases drastically.
For 2018, it’s vital that your business considers the following six security concerns and takes appropriate preventative measures. When adopting cloud computing, or Big Data analytics solutions distributed across a range of environments, security must become a top priority.
1. Securing Trusted Environments
Big Data use is not restricted to the cloud—many organizations, particularly large companies, run Hadoop or NoSQL databases in trusted internal environments. While such environments are useful for combating external security threats, it’s vital to take measures to protect against insider attacks.
After all, it just takes one unhappy employee with access to a server to gain access to a company’s sensitive information and use that information to damage its reputation. However, complicity is not actually required for a successful insider attack, as the case of JPMorgan showed, when an employee’s stolen login credentials were enough to gain access to a server.
Securing trusted environments should incorporate techniques such as anomaly detection, in which an employee’s baseline interactions with Big Data systems are established, and any abnormal behavior such as accessing new directories can trigger alerts for closer scrutiny of such employees. It’s also advisable to use two-factor authentication for access to all servers and systems containing sensitive data.
2. Data Loss & Cloud Backups
It’s imperative to create data backups in on-premise systems because data security does not solely revolve around malicious threats: good data security also protects data from corruption or loss. However, it’s just as important to back up data stored in the cloud, because, after all, cloud computers are still vulnerable to natural disasters or cloud provider errors that result in potential data loss.
Software services exist, for example, that can create AWS snapshots automatically, making the whole cloud backup process extremely simple. If you use any AWS service, such as Amazon EC2 instances to run cloud applications, you can create AWS snapshots, allowing for easy disaster recovery and smooth business continuity. Similar third-party services exist for backing up data stored in other cloud provider systems.
3. Sufficient Access Management
Reading through the examples highlighted at the beginning of this article, you’ll have noticed that most serious breaches in the cloud occur due to incorrect identity/access management. Weak access configurations increase the probability of more successful attacks in addition to increasing the chances of data breaches due to employee errors.
Organizations must enforce a principle of least privileges for employees to minimize the potential damage a compromised account can do. Furthermore, any good identity and access management system must allow IT administrators to manage users across all enterprise systems, both cloud and on-premises, in a consistent, visible, and scalable manner.
4. Performing Due Diligence
Enterprises adopting Big Data and cloud computing because everyone else is doing it face the risk of adopting at such speed that they fail to perform due diligence on their chosen cloud providers.
By not jumping in the deep end, enterprises can examine and evaluate a range of cloud and Big Data solutions and vendors with a checklist of criteria, on which security must prominently feature.
5. Combating API Vulnerabilities
This particular challenge centers on the cloud vendors rather than the users of cloud services. Since cloud providers expose APIs to their users, allowing them to interact with the cloud provider’s services, the vendor has a responsibility to ensure their APIs are secure. Consistent scanning for and fixing of API vulnerabilities will remain a huge factor in the continued success of cloud computing.
6. Security Automation
Cloud adoption and the Big Data economy present a huge challenge for IT staff in continuously tracking data to identify, categorize, and protect highly sensitive information, such as personally identifiable information (PII), personal health information (PHI), and intellectual property.
Modern solutions leveraging innovative technologies such as machine learning can become stalwarts of the cloud computing security landscape by automating the typically labor-intensive task of understanding where sensitive data is stored and the methods used to access it. Such automated security solutions can even monitor the sensitive data and send alerts when suspicious usage patterns are detected, saving real headaches for IT security personnel.
Closing Thoughts
While cloud computing and Big Data offer many positive benefits, they also present some unique security challenges. The truth is that cloud systems are much more secure now than at any point in time, and they continue to become more secure as adoption increases.
However, security is a two-way street, and the users of cloud computing platforms and Big Data systems have just as much of a role to play in data security as the vendors. By taking specific steps, such as to create AWS snapshots for disaster recovery, and following broader guidelines, like performing due diligence, enterprises can ensure the continued security of their data, regardless of the systems on which it’s processed, stored, and analyzed.
The post 6 big data and cloud security concerns to watch in 2018 appeared first on Big Data Made Simple - One source. Many perspectives..
Technology vs credit card fraud is a never-ending battle, in which the stakes are high. Hackers are creating new sophisticated techniques to illicitly obtain credit card information and use it to satisfy their financial needs. To avoid losing billions of dollars, credit card and eCommerce companies, as well as banks, bet on best-in-class software solutions — with big data being touted as the most effective one.
Read on to understand what the beauty of this technology is and why it might become a silver bullet.
Big data is a potent method
It’s better to prevent fraud rather than try to find cures after it has already happened. Big data systems coupled with machine learning can address that.
What does the whole process look like? Analytics systems collect information from a number of sources — customers’ devices, social media accounts, call center conversations, and more — to glean deep insights and create a user profile. Such a profile will include certain behavioral patterns, and any deviation from typical behavior is a reason for raising a red flag.
Here are some typical scenarios of shady activities that should arouse suspicions:
Once the scenario is tagged as dubious, the card provider notifies the customer via a phone call asking to verify the purchases.
Julie Conroy, a research director at Aite Group, a Boston-based advisory firm, admits that many providers are moving towards automated systems for user notification. Instead of phone calls, card holders may get a text message asking if a transaction was really made by them and to respond ‘yes’ or ‘no’.
“Some credit card issuers give consumers the ability to set their own preferences, so if something over $500 hits your card, we’ll let you know… Companies are putting that power in the hands of consumers,” she adds.
There’s room for improvement
There’s more to automated fraud detection than meets the eye. And the decision to block a certain transaction may be a double-edged sword.
Let’s examine the following course of events. A card owner goes overseas without informing the provider of the next location. In this case, there’s a possibility of their transactions being blocked. However, if rightly implemented, your analytics system will instantly analyze a user’s mobile and social media data to detect the exact location and reduce the risk of false positives.
Still, financial companies may face more complicated scenarios. What if card holders change their typical buying behavior for some reasons? Here, the risk of providers incorrectly flagging legitimate purchases is much higher.
If you want to improve fraud detection and avoid causing much discomfort for customers, make sure your system is constantly learning about fresh data and discovering new patterns to deliver rich insights.
What else deserves particular attention?
Data collection is the staple of a fraud detection strategy. So before putting your shoulder on the wheel, check the data privacy rules applicable in your country. With all the necessary information at your fingertips, you will understand what type of data you’re eligible to gather to ensure client data safety.
As the art of fraud is forging ahead, remember to keep track of fresh trends and train your analytics system on new data. In addition, evaluate the success of your strategy to make certain you’re outperforming fraudsters.
The post How to use big data to successfully fight credit card fraud appeared first on Big Data Made Simple - One source. Many perspectives..
The collection of data has grown into a substantial industry in its own right. But what too many small business owners don’t know is that big data can help them out, too — it’s not just for major corporations.
But data is useless without a mission in mind — and what more important mission is there in business than delivering a high-quality experience? If your customers aren’t happy, nothing else about your business can work. So how can you leverage big data to make sure they’re satisfied?
Here are some ideas to get you started if you’re feeling a little lost on the digital seas.
Use Data to Become Proactive Instead of Reactive
The business world has no time for reactive companies. If you’re flying by the seat of your pants instead of anticipating disruptions and investing in plans to answer them, you might discover why only about half of all small businesses survive to see their five-year anniversaries.
Arguably the most important advantage of leveraging big data is the ability to witness patterns over time. Getting real about data collection is like taking the pulse of your business — you’ll have some warning signs if things are taking an unfortunate turn and you’ll probably have some idea of how to right the ship if you study these trends for long enough.
For a look at how data can help us spot and get ahead of emerging trends, just look at how the Centers for Disease Control collaborated with Google to cross-reference search terms and doctors’ reports to see where influenza outbreaks were underway or about to occur.
Think about the advantages of having streams of real-time data on sales and customer trends as they happen. Big data can help you oversee the movement of employees, equipment, merchandise and other assets at every step of your manufacturing process or supply chain.
You’ll be able to see bottlenecks in progress and identify problematic steps in the process which might be slowing you down. None of these insights were possible at this scale — or with this level of detail — just a few short years ago. Now, you can leverage data you didn’t even know you were collecting.
Something like 80 percent of all companies collect data they don’t use — and this is a big loss, because investing in big data-powered analytics could help these companies see their data as a real advantage in a crowded and unpredictable marketplace.
Use Data to Improve Your Efficiency
Part of big data’s draw for businesspeople is the promise of accessing high-level, actionable data from nearly anywhere in the world. Data is mobile-first these days, meaning all of the tracking you can do for supply chains, manufacturing and product shipments can be served from your operational hubs to nearly any mobile device and nearly any employee, no matter where they happen to be working.
For example, giving salespeople access to relevant data while they’re out in the field could result in more than a 60 percent improvement in “sales time” compared with traditional methods.
Big data analytics can help out back at your headquarters, too. Responding to negative customer experiences, for example, used to be a time-intensive process. Now, you can respond faster than ever to specific events which can trigger a response, like a bad review on one of your ecommerce channels or an uptick in bug reports from your application’s latest design iteration.
Real-world examples abound. UPS has collected and collated more than 16 petabytes of data to facilitate more than 39 million parcel tracking requests daily. UPS and similar drivers are under significant time pressures, especially around peak holiday seasons. This massive collection mechanism for logistical data ensures their massive fleet of vehicles and operators wastes as little time as possible responding to unforeseen situations and changes in variables which might delay future shipments.
Another example of using data to reduce operational friction and subtly improve customers’ experiences without their knowledge comes by way of T-Mobile. The cellular provider has reportedly made an art form of performing analysis on billing systems and even social media dispatches to reduce customer defections by 50 percent in just one quarter.
Even processes like search engine optimization and internet PR get an efficiency boost from real-time data and analytics. You can see search trends and link-building in-real time, check it against geographic and demographic data and ultimately help steer the conversation your company has become a part of.
Use Data to Become More Personal and Successful
The business world is beginning to identify a strong correlation between the personality of a company and that company’s success. Among mobile marketers involved in a recent poll, 85 percent indicated their companies enjoy more successful conversions, better customer engagement and stronger revenue streams when they get serious about personalization.
And personalization is fueled by big data. Think of how the world’s streaming services deliver a personal experience: they keep track of what their subscribers are into and deliver additional content that should resonate with their audience. You can do the same by suggesting products that complement their purchases nicely or by sending targeted emails emphasizing testimonials or positive reviews for items they left in their shopping basket.
Put more simply, big data lets your company act more like a human being. You get to build a kind of digital rapport with your audience and give them more of what they want. Out in the real world, the importance of this process can’t really be understated. Given that some products literally mean life and death for customers — as with healthcare — industry and government in Singapore are coming together to leverage data systems to help them better tailor treatment regimens to each patients’ unique blend of health history, family background, genetics and lifestyle. The result is a way to more affordably and effectively practice medicine.
Granted, leveraging personal information requires a tricky balance between usefulness and privacy — a balance too few technology-based companies take seriously. But here’s the thing: if your company already has compelling trust signals telling your customers they can rely on you to use their data only for in-house purposes, this decision should be a no-brainer for them.
As you can see, data powers everything these days. If you haven’t yet had a serious consultation with your company leadership or with a third party about how to put data to work for you, you might be missing out on some significant opportunities to take better care of your customers — and give them something great to talk about.
The post Feel lost in the digital world? Here’s how to use data to boost customer experiance appeared first on Big Data Made Simple - One source. Many perspectives..
A data breach can take a toll on a company of any size. Here’s a look at some significant data breaches, tips to protect your business, and what to do in the event of a data breach.
The post Data breaches are inevitable — or are they? [Infographic] appeared first on Big Data Made Simple - One source. Many perspectives..
Every business now recognizes the power of Big Data Analytics in developing deep actionable insights to enjoy business advantages. However, unlike before when businesses were required to deal with gigabytes of data, the present scenario requires to store and process huge piles of data that is measured in petabytes and terabytes as it is produced by rapidly growing internet population, systems and enterprises. But, as we all have learnt all the years growing up, no problem lasts forever in the technology world. Likewise, Hadoop Analytics is one such tech solution that brings an end to all your big data analytics concerns.
Hadoop is an open-source framework that lets an organization to process huge data sets parallely. Hadoop was designed keeping in mind that system failures is a common phenomenon, therefore it is capable of handling most failures. Besides, Hadoop’s architecture is scalable, which allows a business to add more machines in the event of sudden rise in processing-capacity demands.
As said earlier that the amount of data available today is humongous, the role of Hadoop in big data analytics becomes very important. Hadoop works by filtering and breaking large amounts of data into pieces, and then distributing each piece of data to several nodes of a specific cluster for processing. It’s very important to understand the core of Hadoop if you want to give your business a competitive edge using big data analytics. Many big organisations working on blockchain technology like ethereum gold are also making use of big data analytics to better take care of their data.
Basic Components of Hadoop Architecture
Hadoop Distributed File System (HDFS) : HDFS is the distributed storage system that is designed to provide high-performance access to data across multiple nodes in a cluster. HDFS is capable of storing huge amounts of data that is 100+ terabytes in size and streaming it at high bandwidth to big data analytics applications.
MapReduce: MapReduce is a programming model that enables distributed processing of large data sets on compute clusters of commodity hardware. Hadoop MapReduce first performs mapping which involves splitting a large file into pieces to make another set of data.
After mapping comes the reducing task, which takes the output from mapping and assemble the results into a consumable solution. Hadoop can run MapReduce programs written in many languages, like Java, Ruby, Python, and C++. Owing to parallel nature of MapReduce programs, Hadoop easily facilitates large-scale data analysis using multiple machines in the cluster.
YARN: Yet Another Resource Negotiator or YARN is a large-scale, distributed operating system for big data applications. YARN is considered to be the next generation of Hadoop’s compute platform. It brings on the table a clustering platform that helps manage resources and schedule tasks. YARN was designed to set up both global and application-specific resource management components. YARN improves utilization over more static MapReduce rules, that were rendered in early versions of Hadoop, through dynamic allocation of cluster resources.
Every business has different data analytics requirements, which is why Hadoop ecosystem offers various open-source frameworks to fit your special data analytics needs. Let’s check out below!
Apache Hadoop Frameworks
1. Hive
Hive is an open-source data warehousing framework that structures and queries data using a SQL-like language called HiveQL. Hadoop allows developers to write complex MapReduce applications over structured data in a distributed system. If a developer can’t express a logic using HiveQL, Hadoop allows to choose traditional map/reduce programmers to plug in their custom mappers and reducers. Hive is a very good relational-database framework and can accelerate queries using indexing feature.
2. Ambari
Ambari was designed to remove complexities of Hadoop management by providing a simple web interface that can provision, manage and monitor Apache Hadoop clusters. Ambari, which is an open-source platform, makes it simple to automate cluster operations via an intuitive Web UI as well as a robust REST API.
Ambari’s Core Benefits:
3. HBase
HBase is an open-source, distributed, versioned, non-relational database model that provides random, realtime read/write access to your big data. Hbase is a NoSQL Database for Hadoop. It’s a great framework for businesses that have to deal with multi-structured or sparse data. HBase makes it possible to push the boundaries of Hadoop that runs processes in batch and doesn’t allow for modification. With HBase, you can modify data in real-time without leaving the HDFS environment.
HBase is a perfect fit for the type of data that fall into a big table. HBase first performs the task of storing and searching billions of rows and millions of columns. It then shares the table across multiple nodes, paving the way for MapReduce jobs to run locally.
4. Pig
Pig is an open-source technology that enables cost-effective storage and processing of large data sets, without requiring any specific formats. Pig is a high-level platform and uses Pig Latin language for expressing data analysis programs. Pig also features a compiler that creates sequences of MapReduce programs.
The framework processes very large data sets across hundreds to thousands of computing nodes, which makes it amenable to substantial parallelization. In simple words, we can consider Pig as a high-level mechanism that is suitable for executing MapReduce jobs on Hadoop clusters using parallel programming.
5. ZooKeeper
ZooKeeper is an open-source platform that offers a centralized infrastructure for maintaining configuration information, naming, providing distributed synchronization, and providing group services. The need of a centralized management arises when a Hadoop cluster spans 500 or more commodity servers, which is why Zookeeper has become so popular.
ZooKeeper also avoid single point of failure situation as it replicates data over a set of hosts, and the servers are in sync with each other. Although Java and C are currently used for ZooKeeper applications, Python, Perl, and REST interfaces could also be used someday for ZooKeeper applications.
Apache Hadoop is a great platform for big data analytics and there are various other technologies, like NOSQL, Avro, Oozie, and Sqoop, available in the tech market that makes Hadoop ecosystem very versatile. You must carefully assess big data analytics needs of your business before jumping on a Hadoop technology, so that you get the best results. Hadoop big data analytics has already helped many businesses to touch new heights, you can also help your business grow by using Hadoop analytics for data science.
The post Basic components of Hadoop Architecture & Frameworks used for Data Science appeared first on Big Data Made Simple - One source. Many perspectives..
