What is ArgoCD?

ArgoCD is known as a declarative GitOps tool that is based on Kubernetes. Let's try to break this down. "GitOps" is a process where you take the code you have written and pushed to the Git registry (e.g., GitHub, GitLab, Bitbucket) and take it to the deployment, which is mainly done and implemented as automation. The word "declarative" means that the deployment has the same architecture that you want and have decided on. So if there was a case that something gets changed in the deployment then it will revert to the previous state (the one which you have mentioned)

CD workflow without ArgoCD:

So whenever the application is pushed to the git repository, there are certain steps it goes through to reach the deployment. These steps are automated and known as the "CI/CD pipeline." So once the code changes are pushed, they are tested, the image is built, pushed to the DockerHub or other registry, the manifest files are updated, and finally, through kubectl apply, it is applied to the Kubernetes cluster.

But in this case, we have to face and resolve challenges like:

• Install and setup tools like kubectl

• Configuring access to Kubernetes

• Configuring access to could providers

• Several security challenges

• No visibility of deployment status

CD workflow with ArgoCD:

So in a general scenario, the above-mentioned workflow is used, but while working with Kubernetes, we might need to change this workflow. It is essential to know that according to the established best practices for the Git repository, we should have two separate repositories for code files and application configuration (K8s manifests).

The main advantage is that there can be a time when we want to change only the config files but don't want the test -> build -> run workflow to run, as the configs can be adjusted independently of the application built. Having this as a reference now, ArgoCD (which is installed on your Kubernetes cluster) keeps track of the state you mentioned in the configuration repositories.

So if were to use the earlier CI/CD pipeline, then it would have been too complex to manage and configure. Once the application is built and up the changes are reflected in the configuration files. These changes are detected by ArgoCD and pulled into the cluster.

Note: ArgoCD works on the pull method and not like the usual push method

Advantages of using ArgoCD:

• Git as a single source of truth:

  This means that if any changes are to occur in the Kubernetes cluster then the git repository is a place to validate and check if everything is going as mentioned in the manifests and any instance that is out of instructions is rolled back.

• Easy rollback:

  This is a very essential feature of ArgoCD as it helps us to revert to the previous state if there was any problem or misconfiguration in the cluster.

• Cluster disaster recovery:

  This means if one of my servers were to go down then I can create another cluster with ArgoCD and point it to the same repository then the process for recovery becomes much more efficient and fast

• K8s access control:

  We can manage the cluster access directly through git so we don't have to worry about giving access to the cluster to an external tool.

Let's try it out!

ArgoCD is super easy to install and configure. We will be looking at it through an example. I will be using Minikube for my Kubernetes cluster.

I will be using this repository for this demo purpose and you can use the same

1. Install ArgoCD into your cluster using the following commands

   kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

1. Now that we have installed the good in the argocd namespace we can now create a yaml file with the following code. I will name the YAML file as application.yaml.

    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: wmd-argo-application
      namespace: argocd
    spec:
      project: default
   
      source:
        #Your repo link here
        repoURL: https://github.com/MaungNoorIslam/-practical-devops
        targetRevision: HEAD
        #Location where the manifests are stored
        path: ArgoCD/dev
      destination: 
        server: https://kubernetes.default.svc
        namespace: wmd
   
      syncPolicy:
        syncOptions:
        - CreateNamespace=true
   
        automated:
          selfHeal: true
          prune: true
   

2. Create a dev folder and place your deployment and service files into it like below. (Notice that it's the same path that I have mentioned in the above application.yaml file.

1. Let's create a namespace for our application as we have mentioned in the manifest files. (Again the manifest files like deployment.yaml and service.yaml are already present on the repo so you can check them out there.

# You can give any name to your namespace and update respective files
kubectl create namespace wmd

1. Now we have to apply the application.yaml file to the cluster. We can do that using the kubectl apply command below

kubectl apply -f application.yaml

1. Let's go and check out it on the ArgoCD dashboard. To see the dashboard run the following command on your terminal.

    kubectl port-forward svc/argocd-server 8080:443 -n argocd
   

   and now you can go to http://localhost:8080 to access the ArgoCD dashboard

   1. In the Username field enter admin and to generate a password run the following commands into the terminal.

        kubectl get secret argocd-initial-admin-secret -n argocd -o yaml
      

      and you will get output something like:

   2. 

We are not done yet, copy the password in this case mine, will b

e VktPYkNUZGZRcjlNVk1ibQ== now put this password into another command to decode it and obtain the real password. Run the following command to decode it.

# Change after echo with your password
echo VktPYkNUZGZRcjlNVk1ibQ== | base64 --decode

Now copy the password that is obtained (Remember to omit % at the end of the password obtained) and put it into the ArgoCD dashboard

1. Now you will be able to view the application in the dashboard.

Click on the application name and you are able to see a beautiful map of your deployment.

You can see every possible information about the cluster here including the health of the pod, cluster and service. ArgoCD pulls the changes every 3 minutes this can be changed using webhooks and other methods.

And done 🎉 you have successfully configured and implemented ArgoCD to your cluster now whenever you want to change anything in the deployment then you can directly change the manifest and push the code to the git repository and changes will be reflected in ArgoCD automatically.

Resources:

• ArgoCD documentation

• ArgoCD Tutorial for Beginners | GitOps CD for Kubernetes

• What is ArgoCD

Thank you so much for reading 💖

Like | Follow | Subscribe to the newsletter.

LinkedIn: https://www.linkedin.com/in/maung-noor-islam/

What is Prometheus 🤔

Prometheus is used as a monitoring tool for highly dynamic container environments or even bare metal servers. Okay, so with the increasing complexity of applications, it is becoming increasingly difficult to handle everything manually and necessitates automation. For example, if you have many services and objects running on the server, there is no insight into what is going on at the hardware level or application level. There might be cases when a service goes down and that causes other services to malfunction. In such cases, you should be able to quickly find the cause and solve it.

In a normal scenario, it won't be easy because you'll have to go a long way back to find the service at the root of the problem and fix it. So Prometheus constantly monitors all the services and alerts the admins or individual users when crash. Prometheus also helps you to detect these problems beforehand (eg: resource depletion, memory shortage, ram/ memory usage)

Prometheus Architecture ✨

Prometheus server:

This is actually responsible for monitoring the work. Prometheus server is comprised of 3 different components

• Data Retrieval Worker: This component pulls the metrics data from the application and services

• Time Series Database: This is a database that saves all the matrix data.

• HTTP Server: This component accepts the queries and provides the data from Time Series Database to The web UI (eg: Grafana)

(You can see how these components are put in architecture from the image below)

What are Targets 🎯:

Targets are any objects that are monitored by Prometheus. These targets include Linux or Windows servers, Apache servers, single applications, services, or databases. And these targets have units for monitoring. For eg units for Linux servers can be CPU states, Memory usage, etc.

What are Metrics 📊:

Prometheus provides a human-readable format for metrics collected from the targets. Metrics entries are separated by the type and help attributes. HELP attribute is used to describe what the matrics are and there are 3 types of metrics

• Counter: These metrics keep track of how many times a particular error was caused or alerts were sent.

• Gauge: This metric keeps track of the current value of the unit.

• Histogram: This matrix keeps track of how long and how big the size of the request was.

Target endpoints and exporters 🚛:

Some services have a default endpoint exposing data to Prometheus but many services need another component, and that component is an exporter. The exporter is a service or a script that fetches the data from the matrix, converts them to a format that Prometheus understands, and exposes this converted data on its own /metrics endpoint.

Advantages of Alert Manager 🔔:

An alert manager is used to check the rules set by the admin or the user and triggers when any of the given rules is reached. The alert manager then sends the alerts and signals on the provided channels like email, slack, discord, etc.

Common characteristics of Prometheus 📝:

• Reliable: Prometheus is highly reliable as you are able to get a clear insight into what is going on inside your services and application

• Stand-alone and self-containing: Prometheus doesn't need any outer service to support its functioning.

• It also works even if other parts of the infrastructure are broken. It is meant and supposed to work as a separate service.

• Prometheus doesn't need any extensive setup for implementation. It has its own helm chart which makes it absolutely easy to configure and use for monitoring your infrastructure and applications

• Prometheus is less complex compared to other monitoring tools.

Setting up Prometheus on K8s ⚙️:

1. Create a separate namespace called monitoring

   kubectl create namespace monitoring

2. Add Prometheus repo to helm:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

3. Update the helm repo list

   helm update repo

4. Install Prometheus operator on your cluster using helm:

   helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

5. Port-forward Grafana dashboard service to access the Prometheus metrics:

   kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

6. Visit http://localhost:3000 to access the Grafana dashboard

7. Once on the dashboard, you can use the following username and password to log in as they are the default for everyone at the start and of course you can update them if you want.

   username: admin

   password: prom-operator

And that's it; you have successfully added Prometheus as a monitoring engine for your infrastructure and modified the Grafana dashboard according to your will and preference to suit your needs. The most essential parts of Prometheus are time series data and a time series database, which contains the types of metrics you can expect. If you delve deeply into the preceding topics, you will gain a much better understanding of how you can modify Prometheus at each step to suit your specific use case.

References 📖

• Prometheus Documentation

• Setup Prometheus Monitoring on Kubernetes using Helm and Prometheus Operator

• Writing a Prometheus exporter from IDE to deployed in 20 minutes

• How Prometheus Monitoring works | Prometheus Architecture explained

• kaiwalyakoparkar/practical-devops

Thank you so much for reading 💖

One of the main advantages of cloud computing is cost savings. Instead of investing in and maintaining costly physical servers and IT infrastructure, businesses can pay for the resources they use on a subscription or pay-per-use basis. This not only reduces the initial investment required but also eliminates the ongoing maintenance costs associated with managing physical hardware.

Scalability is another benefit of cloud computing. Businesses can quickly and easily scale their IT resources up or down depending on their needs. This is particularly useful for businesses with fluctuating demand, such as those that experience seasonal spikes in traffic or sales. With cloud computing, businesses can scale their IT resources in real-time, ensuring that they have the resources they need to meet demand without overprovisioning.

Flexibility is also a key advantage of cloud computing. By accessing resources remotely, users can work from anywhere with an internet connection, enabling businesses to offer remote work options and facilitating collaboration across geographically dispersed teams. This can improve productivity and help businesses attract and retain top talent.

However, there are also some potential drawbacks to cloud computing. One concern is security, as sensitive data and applications are stored remotely and accessed over the internet. Businesses must ensure that appropriate security measures are in place to protect against data breaches, cyberattacks, and other security threats.

Another potential issue with cloud computing is vendor lock-in. As businesses become more reliant on cloud services and infrastructure, they may find it difficult to switch to a different provider or bring IT resources in-house. This can limit flexibility and increase dependence on the cloud provider, which may impact pricing and service levels.

Overall, cloud computing has become an essential part of modern business operations. While there are potential drawbacks to consider, the benefits of cost savings, scalability, and flexibility make cloud computing an attractive option for businesses of all sizes and industries.

#business #software #sales #work #data #experience #cyber #investment #security #cloud #collaboration #cloudcomputing #help #productivity #investing #talent #infrastructure #maintenance #remotework #hardware #scalability

Many users within an organization play important roles in the machine learning (ML) lifecycle. There are product managers, who can simply type natural language queries to pull necessary insights from BigQuery, data scientists, who work on different aspects of building and validating models; and ML engineers, who are responsible for keeping the models working well in production systems. Each of theses involves different needs, and this post covers the Google Cloud M and AII services that are available to help meet those needs.

Pick your AI/ML Path The services that will work best for you will depend on your specific use case and your team's level of expertise. Because high-quality ML models require a significant amount of effort and ML expertise to build and maintain, a general rule of thumb is to use pretrained models or AI solutions whenever possible – that is, when they fit your use case. If your data is misstructured and in BigQuery, and your users are already comfortable with SQL, then choose BigQuery ML. If you realize that your use case requires writing your own model code, then use thee custom training options in Vertex AI. Let’s look at your options in more detail.

Prepackaged AI Solutions Both pretrained APIs and prepackaged AI solutions can be used without prior ML expertise. Here are three prepackaged solutions that can be used directly: Contact Center A:- Create rich and natural conversational experiences across devices and platforms using an AI-powered virtual agent, insights derived from customer interactions, and agent assist features.

Document A:- Tap into your unstructured data (such as images and PDFs) and make it accessible using Google computer vision (including OCR) and natural language processing (NLP) capabilities to increase operational efficiency, improve customer experiences, and inform decision-making. Recommendations AI - U: machine learning to deliver recommendations personalized for each customer’s tendencies and preferences across all touchpoints.

Pretrained APIs If you don’t have any training data to train a model and you have a generic unstructured data use case such as video, images, text, or natural language, then a pre-trained API would be a great choice for your AI/ML project. Pretrained APIs are trained on a huge corpus of generic, unstructured data that is built, tuned, and maintained by Google. This means you don’t have to worry about creating and managing the models behind them

Vision AI - Derive insights from your images in the cloud or at the edge with Auto ML Vision or use pretrained Vision API models to detect emotion, understand text, and more. Video AI - Enable powerful content discovery and engaging video experiences. Translation AI - Make your content and apps multilingual with fast, dynamic machine translation.

Language AI - Derive insights from unstructured text using natural language understanding (NLU). Get insightful text analysis that extracts, analyzes, and stores text. Speech-to-text API - Accurately convert speech into text (and vice versa with Text-to-speech API) to deliver a better user experience.

BigQuery ML If your training data is in BigQuery and your users are most comfortable with SQL, then it likely will make sense for your data analysts and data scientists to build ML models in BigQuery using BigQuery ML. You will have to make sure that the set of models available in BigQuery ML matches the problem you’re trying to solve. BigQuery ML offers simple SQL statements to build, train, and make predictions within the BigQuery interface or via the API.

Vertex AI Vertex AI offers a fully managed, end-to-end platform for data science and machine learning. If you need to create your own custom models with your own data, then use Vertex AI. Vertex AI offers two options to train models: AutoML and custom training. Here is how to choose between these two options:

Use case: If your use case fits a supported AutoML offering, then starting with AutoML is a good choice. This includes use cases involving data types such as image, video, text, and tabular. But if your model takes a mixed inputtype,e such as images and tabular metadata, then it makes sense to use a custom model.

Requirements: If you need control over your model's architecture, framework, or exported model assets (for example, if your model needs to be built with Tensor Flow or Pytorch), then use a custom model. Team expertise: How experienced is your team with machine learning and /AI? If you have a team with limited experience in building custom models, then explore Auto ML before you look into custom model development.

Team size: If you have a small data science and ML team, then it may make more sense to work with Auto ML because custom model code requires more time to develop and maintain. Prototyping: Use AutoMLL if you want to develop a quick initial model to use as a baseline. You can then decide if you want to use this baseline as your production model or look to improve upon it by developing your own custom model. For a more in-depth look into Service Directory, check out this documentation or start with the detailed Vertex AI videos. Also, join us on June 9th at a.m. PST for the Applied ML Summit to learn more about the process of successfully shipping machine learning models into production..

As an example of this process, imagine that you wanted to predict the water level of lakes in your area. To do so, you collect metrics on rainfall and lake water levels for all of the lakes in your area. magine further that you were able to get this same data collected over the course of several years. Using this data we could develop a model that would allow us to predict what the water level in the lake will be based upon the amount of rainfall that is received.

Regression is a process used to develop a model that rationalizes the data and identifies predictive trends within the data. There are a number of different types of regression models that can be used to predict different types of relationships. Some of these relationships, when plotted, are linear; others have a curved shape. We will walk through an example of the simplest form of regression, which is linear regression, which is used when a scatterplot of the data has roughly a linear shape. Going back to our lake level example Let us assume that we have all of the data as follows.

If we look at this graph, one of the things that we realize is that the plotted values begin to form a pattern. In fact, we could draw a straight line that follows the shape of the data points that have been plotted. In figure (x), we added a line to the graph to indicate the general shape of the points. This line is called a regression line. Essentially, this line provides us with a way to use the data to create a prediction model. For example, if we look at the graph, we can see that, in general, when we have 15 inches of rainfall, the lake levels will tend to be 2 inches lower. We can also see that when we have 20–25 inches of rainfall, the levels of water in the lakes tend to be 3 inches higher.

In the figure, we just kind of "drew" a line on the data, but the regression algorithm employs a process to "fit" the line to the data. The first thing that we need to understand about fitting a regression line is the concept of error. When we look at Figure (x), we can see that many of the data points do not touch the line. The line represents the predicted value of Y for any input value of X. The dots that do not touch the line indicate that the actual data doesn’t match the prediction.

Linear regression is clearly not going to be an accurate solution to predict these values. What is needed is the ability to have a curved line that models the behavior of the data to increase up to a point and then decrease. Fortunately, we have such a solution in curvilinear regression, which uses splines to model data that is not linear yet has a strong relationship between the data attributes. In curvilinear regression, we no longer have an algorithm to fit a "line," but rather an algorithm to fit a "spline," which is accordingly called "spline fitting."

The reason that we square the value is because the values that are above the red line will have a ‘positive’ value and the points below the line will have a ‘negative’ value. When we square them, all of the distances become positive. The algorithm must then minimize the value when all of these ‘errors’ have been added together.

Supervised learning for prediction The lake's water level is an example of a linear regression. We call it linear regression because it fits a straight regression line to the data, and this line is expressed in the form:

y = mx + b

If you recall your basic algebra, this is essentially a function where for any input value of x, there is an output value of y. The m in the equation is called the slope. The slope defines the rise over run or the change in the value of y for any value of x. The b in the equation specifies the y intercept, or the point where the line intercepts the y axis. The y intercept is typically the point where the value of x is zero.

Curvilinear regression There are other forms of regression that are often important and needed. For example, consider the following data set (Figure x). The data in this graph are clearly not linear. We see the value in Y rise with the values in X up to a point around 4 on the X axis, and then they begin to decline.

Decision-making algorithms ingest problem-relevant information from the environment and produce an action. Think of an AI algorithm that takes in patient vital signs and outputs a diagnosis or a stock-trading system that synthesizes daily market prices and suggests stock buys.

But building an agent in a highly uncertain environment is a challenge for any developer. Kochenderfer and his co-authors recommend various approaches for designers to solve various types of problems in this new book. For example, if a designer knows that there’s a particular type of uncertainty in the environment, or that a source of input, such as a sensor, is imperfect, then they would go to a particular part of the book, and it would outline a number of different algorithms they might use.

Here, Kochenderfer discusses the value of computer-based algorithmic decision-making and the key themes of the book.

When do computer algorithms make better decisions than humans?

Humans are not very good at reasoning about low probability events or about complex scenarios where many things are happening at once. That’s where a computational approach can bring a tremendous amount of value to a decision-making process. It reduces the burden on the human designer to anticipate all the possible scenarios.

For example, a human designing a self-driving car cannot anticipate every possible driving scenario or the kinds of sensor failures and errors that might arise when things go wrong.

A related example — we have studied aircraft collision avoidance systems that make decisions using a process called dynamic programming that can reason about very low probability events, such as unexpected maneuvers, and optimize for the best possible course of action given the various sources of uncertainty. Rigorous analyses showed that these collision avoidance systems are both safer and more efficient than something that a team of humans could have produced on their own.

Why is decision making under uncertainty a particular focus of the book?

Most of the decisions we make in our lives are based on imperfect information. When we make a medical decision, for example, we know that diagnostic tests might be imperfect — there can be false positives or false negatives; and when we’re building robots, the sensor systems might fail in some way; or a self-driving car might encounter occlusions in the environment, such as a van blocking our ability to see a pedestrian.

We’re just inherently uncertain about the state of the world and sometimes that uncertainty is a significant factor. So, we want to address problems with uncertainty head on, and that’s a key aspect of decision making that this book is trying to address. We want to help people build decision-making algorithms that can take imperfect information and make decisions that achieve an objective or set of objectives.

And when we talk about uncertainty, that includes uncertainty about the effects of our own actions, uncertainty about the state of the environment, uncertainty about how others might respond to our actions, and uncertainty in our conception or “model” of how the world works.

And the book breaks these forms of uncertainty down to their essence — to very simple computations. And it turns out that computers can do these computations pretty easily using multiplication and addition of potentially small numbers.

How does time play a role in algorithmic decision making?

Time is critical. We generally need to reason about the effects of our actions over an extended time window, including keeping track of the recent past as well as making predictions about the future. Most real-world problems don’t involve single-shot solutions.

In the book we start off by introducing probability theory and utility theory in single-shot contexts so that the reader gets a solid understanding in this more simplified context. But we then move on to sequential problems, because decision makers typically want to reach a goal that is going to require a series of actions. For example, in a medical context, doctors don’t make a single decision and that’s it. They hopefully have a long-lasting relationship with the patient, so we don’t want an algorithm to greedily make what appears to be the best decision in the moment. We need it to reason about the future.

How have various disciplines contributed to the field of algorithmic decision making?

Many different communities inspired the content of this book. There’s not only AI, which has traditionally been a subfield of computer science, but also operations research, control theory, psychology, neuroscience and economics. All of these fields have contributed to the concepts in the book in a major way.

In fact, economics is the first one that comes to mind. In the 1940s, John von Neumann and Oskar Morgenstern published a book called Theory of Games and Economic Behavior that sets forth a set of axioms about rational preferences. These are properties that we just accept, such as if I prefer apples to bananas, and bananas to cookies, then I’d better prefer apples over cookies. And their work gave support for the idea of utility theory, which says that so long as you have these rational preferences, you can assign utilities — numeric values — to different outcomes. And that allows you to make the problem of decision making under uncertainty well defined: One need only choose the action that maximizes your expected utility. That’s the maximum expected utility principle, and that principle, which comes from economics, underlies the entire book.

How are you seeing algorithmic decision making benefiting or harming society?

The book highlights several examples of beneficial deployment of algorithms. For example, because of aircraft collision avoidance systems, we will have safer and more efficient air transportation. In the financial sector, algorithmic decision making can help people invest their resources so that they can have a sustainable level of consumption across their lifetime. And medical decision support systems can help promote safer, better medical care.

There are also aspirational kinds of research that have not yet been deployed, such as figuring out how to fight wildfires. Firefighting resources are finite and there’s a lot of uncertainty about exactly how a fire will develop depending on the wind, the vegetation, the terrain and so forth. Algorithms that account for these uncertainties could help us fight fire more effectively and safely.

On the other hand, there are some potential pitfalls. If these systems are deployed without proper validation, there can be a risk to life, and there could be unfairness and bias. So, a major focus of our research is to not just build systems that are worthy of our trust, but to come up with methodologies to validate that they will behave as expected or as desired when deployed in the real world. We want to proactively make sure that these systems are safe and that they have the desired societal impact. And because we want to understand potential issues with our systems well before they are deployed, this book includes a chapter that talks about validation — a topic that is actually worthy of an entire book that we’re currently writing titled Algorithms for Validation.

Let me provide an example. A short while back, I was in Palo Alto, California, on business. I was using the GPS navigation system built into my Android smartphone because I didn't know the roads in Palo Alto very well. The navigator plotted a path to get to my destination and indicated the path with a blue line. In some areas, the blue line turned yellow, and in a few others, it turned red. The yellow sections indicated that traffic was slower than posted speeds, and the red sections indicated that traffic along those sections had slowed to nearly a stop. This color coding of blue, yellow, and red is an example of machine learning in practice.

In another example, I logged into my Netflix.com account to watch a movie. As I logged in, I was presented with a number of movies that were "selected" or "recommended" for me. Each of the selected films is very much like other films that I have watched and enjoyed. It was like someone had picked out these movies just for me. The Netflix system that recommends movies is another example of machine learning.

In a final example, I recently installed an application on my Android phone called SoundHound.Have you ever had a situation where you heard a song being played on the radio and just couldn’t remember the name of the song or the artist who performed it? Using SoundHound, the application "listens" to the song for just a few seconds and then identifies the name of the song and the artist who performed it. It further will show a transcription of the lyrics of the song … as they are sung!

Each of the preceding is an example of machine learning algorithms. Mitchell (2006) defines machine learning as a way to answer the following question: “How can we build computer systems that automatically improve with experience, and what are the fundamental laws that govern all learning processes”

If we consider Mitchell’s question there are a couple of key points. The first is the idea of a computer system that can improve with experience. Consider the Netflix example; no one programmed the computer with a set of preferences for movies that I would enjoy. What did happen, is that Netflix has a machine learning algorithm that looks at the movie selections of many different people, and using this data, it can begin to group those people into different categories? Once placed into one (or more) of these categories the system merely suggested that the movies that others in the same category are watching will be good recommendations for me as well.

What Netflix has done is develop an algorithm that can categorize people into different groups and use these categories to recommend what movies they might enjoy watching. Is the Netflix algorithm learning? What is learning after all?

There is considerable academic debate about the nature of learning. On one hand, we can point to the acquisition of facts or information as learning. In other contexts, the ability to gain knowledge, skills, or the ability to solve particular sets of problems is offered as evidence of learning. Finally, there is an argument that learning involves awareness and that learning is only present if there is awareness of learned processes. In other words, there must be an understanding of what has been learned, learning must be accompanied both by understanding and meaning.

The concept of learning is obviously a complex one, but we are only interested in the concept of learning as it applies to a computer system. Fortunately, machine learning is easier to define than learning in general. Machine learning is a process that uses data to fit a model that can be used to solve particular problem sets and to improve its performance of those problem sets over time as more data and experience are accumulated.

Data Mining and Machine Learning Although data mining and machine learning, have some similarities and share many of the same procedures and algorithms they are different. The key difference between data mining and machine learning is in their intent and application. Data Mining is used to gain insight or knowledge from existing data. The objective of data mining is to get more value (or new value) out of a data set. To this end, there is a wide spectrum of techniques both statistical and visual that are used to provide new views or insight into a data set. Some of these approaches include ()Deshpande & Thakare, 2010):

• Description • Reporting • Visualization • Prediction • Classification • Estimation • Clustering

Machine learning, on the other hand, is designed to develop automated engines that can facilitate decision-making by ‘training’ algorithms to perform a variety of decision tasks. The goal of machine learning is to create an algorithm that has the capability to improve its performance over time. Programs, in general, are sets of instructions that the programmer has created that react to specific data or inputs. The way that the program responds to any input is always the same because the responses are always defined by the individual who created the program. In machine learning, the goal is to develop an algorithm that has the ability to alter its response to some input data. The machine learning algorithm adjusts its response based on the input data thus making itself capable of adjusting how it interacts with its environment. This ability to self-adjust can be defined as a form of learning.

Machine learning algorithms have been found to be very effective for a number of different tasks some examples include the following:

• select the preferred lighting of a room, classify objects,
• recognize specific patterns in (streams of) images,
• identify the words in handwritten text
• understand a spoken language,
• control systems based on sensor data,
• predict risks in safety-critical systems,
• detect errors in a network,
• diagnose abnormal situations in a system,
• prescribe actions or repairs, and discover useful common information in distributed data

The way that machine learning algorithms work, is that a set of training data is used and the machine learning algorithm evaluates the training data to build a model. As more training data becomes available, the model can be improved. What the learning algorithm attempts to do is to develop a set of procedures that fit the training data.

If you have a smartphone then you probably have experienced a machine learning algorithm firsthand. Most current versions of smartphones have voice recognition capabilities. The way that voice recognition works are that a number of samples of words are collected the spoken word is associated with its written form. For example, we might have a sample with eight or 10 different people speaking the word dog and associated with each of these audio files with the spoken word dog we would associate the text form of the word dog.

Each person will say the word dog slightly differently. There will be changes in intonation, pitch, speed, and other factors. What the learning algorithm will do is build a model of what the word dog spoken is like and then attempt to compare a spoken word with this model of course it needs to compare this not only with the word dog but pretty much all of the words spoken in the language using the training data as a base, the machine learning algorithm will determine what word is most similar to the one that was spoken.

In this way even though the system doesn’t have a specific example of you speaking the word dog it can typically identify when you have spoken the word dog because it compares it with all of the training examples that it provided. As the algorithm gets more information it builds better models and becomes more accurate over time. We refer to this ability to get more accurate over time as more data is evaluated as learning. A key feature of machine learning is the fact that the algorithms must have the ability to ‘learn’ over time and as the algorithm is exposed to new information, it has the ability to incorporate the new information to improve its own performance (Larose, 2005).

Types of Machine Learning The discipline of machine learning has developed to include many different algorithms and techniques. These techniques, however, can be grouped into three categories of machine learning Supervised Learning, Unsupervised Learning, and Reinforcement learning.

By "structured," we mean that each piece of data is well-known and understood and fits into a set of guidelines about its format and size. In a relational database, the data values in the tables are often "related" to values in other tables to minimize data redundancy. For example, imagine you were Amazon with millions of customers for whom you processed orders. You wouldn’t want to store the customer’s information about shipping address, preferences, payment methods, etc. In the orders table because most customers place several or even many orders, having to store the same information over and over again each time a customer placed another order would be both inefficient (wasting space) as well as difficult to manage (if the customer moves, then you wouldn’t want to go back and find every order shipped to the customer and update the shipping address information).

To make sure that this "related" data was kept consistent, these relational database systems adopted a set of principles for how to manage changes or additions to the data to make sure that all data was kept consistent at all times. Of course, by ensuring consistency, these database systems also created a lot of complexity and limited their scalability.

Computer scientists figured out that it would be just about impossible to keep the data consistent if the data were to be spread across multiple servers, so they decided to build a database that ran on a single server, and then they attempted to make that single server as large as possible so that it could accommodate large databases. This worked for a while, but as both the volume of data and the variety of data types have increased, the single-server database approach is reaching the end of its capabilities, and for many applications, it is no longer capable of accommodating the required data workload.