Have you ever wondered how a Banks are able to detect anonymous transactions or ever observed Instagram or Facebook detecting uneven activity and temporarily blocking the account from usage?
Yes, it is possible for the devices to do that using a technique called Anomaly detection.
Anomaly Detection:
Anomaly is something which is not usual, abnormal, peculiar or in simple words, it is not normal, or regularity is missing.
however, Anomaly detection is nothing but detecting a change by continuously monitoring the activity of a person, account or anything. It is a machine learning technique to detect abnormal activities and avoid any serious outcomes.
Understand Better:
Asma is a traveller. She spends a lot and pays her daily bills using her credit card. But suddenly, one day a hacker attacked her system and stole all her personal details including the credit card data. thus, The very same night he used the stolen data and transferred a huge amount of data to his account. Asma may be sleeping but the software isn’t, so it detected unusual activity in her account and flagged it for review.
The bank employee then called her and enquired about the transaction. then, She was shocked and informed she didn’t do it. Thus, the bank employees reverted the transaction and helped her change the passwords.
But how did the software detect it?
It is nothing but Anomaly detection. Asma usually does her transactions during the day and for a small amount. But the hacker attempted to perform a transaction of quite a big amount at night which is different from the usual trend. Hence the software flagged it.
There are many ways to detect an anomaly:
- Density-based technique
- Subspace detection for high dimensional data
- One-class support vector machines
- Replicator neural networks
- Bayesian network
- Hidden Markov models
- Cluster analysis based detection
- Deviation from association rules
- Fuzzy logic-based detection
- Ensemble techniques, using feature bagging, score normalization and different sources of diversity
however, Each method is selected based on different parameters available and dataset availability and each has its own systematic advantages over others and is chosen based on the use.
So, in this article lets learn Z-Score technique which is a Density-based technique using an example. thus This algorithm analyses the behaviour based on past data. So, as time passes by, more data gets accumulated and this algorithm works better and gives accurate results.
We will also have a Bolt Wi-Fi module which continuously collects data from the LDR sensor and processes it to find the updated threshold based on the current data. By updating the threshold we are able to predict the present threshold and detect any sudden changes to alert the user accordingly. thus, The built prototype monitors the brightness of the room and if it finds any sudden increase in brightness it alerts the user through SMS and turns off the light(if it is ON). Similarly, if there is any sudden decrease in brightness, it alerts the user with a different message and turns on the light.
SOFTWARE AND COMPONENTS REQUIRED:
| s.no | Component | Type | Setup |
| 1 | Bolt cloud | software | Click here |
| 2 | Bolt Wi-Fi module | Hardware | – |
| 3 | LED | Hardware | – |
| 4 | LDR sensor | Hardware | – |
| 5 | Twilio | Software | Click here |
HARDWARE CONNECTION:
Connections are quite simple.
- First, power up the Bolt Wi-Fi module by a USB cable.
- Now, connect the positive leg of the LED to any of the Digital GPIO pins of the Bolt WiFi module and the Negative leg of the LED to “GND”.
- Then, connect the LDR sensor between “A0” pin and “3V3” pin.
- For limiting the current and to find the voltage across LDR a resistor is connected in series to LDR, between the “A0” and “GND” pins.
CODE EXPLANATION:
The code is written using Python language.
Pre-requisites:
- Python 3 – Download here (While installing, add the path to environment variables and install pip too.)
- boltiot library for python (Enter “ pip install boltiot “ in command prompt. It automatically downloads the boltiot library. The pip installer should be installed priorly for this command to be executed)
1. Import required libraries
- Credentials (A python file containing all required credentials)
- Bolt, Sms library from boltiot module
- json, time library
2. Then, Initialize required variables in the code
- sensor_data=[] //Array to store past data
- min_val=1024 //Minimum value of the threshold
- max_val=0 //Maximum value of threshold
- LED=1 //variable to show our LED pin selected
- last_status=0 //status of our LED (1 or 0)
- data_count=10 //It is the frame size used for our prediction
- factor=0.5 //factor for z-factor prediction
(The z-factor should be less than 1 and greater than 0.5 for most desirable result)
3. Firstly, we link our code with the Wi-Fi module, sms(twilio) module, and initialise the state of LED as LOW.
- Bolt API and Device Id (details from your Bolt cloud)
- Twilio SID, AUTH, To number, From number (details from your Twilio account)
4. Run a continuous loop in which our algorithm runs with a delay time of 10 seconds.
5. In every loop, we have to read the sensor value and get value from the json string obtained. successfully obtained, we will proceed further or if the data fetching is failed, we will print ‘error’ and continue with the loop.
- { “value” : “699” , “success” : “1” } //Json format received
6. If enough data value for the computation is collected, the threshold is calculated. And If not, more data is collected.
or If the ‘sensor_value’ is within the limits, then the new threshold is computed and moves to the next value. But, if it is not within the limit, then we act accordingly, carry out appropriate actions and then move to the next loop.
Code:
The whole code looks as shown below:
import credentials
from boltiot import Bolt
from boltiot import Sms
import json
import time
sensor_data=[]
min_val=1024
max_val=0
LED=1
last_status=0
data_count=10
factor=0.5
mybolt=Bolt(credentials.bolt_api,credentials.bolt_id)
sms = Sms(credentials.twilio_sid, credentials.twilio_auth, credentials.twilio_tonumber, credentials.twilio_fromnumber)
mybolt.digitalWrite(LED,’LOW’)while(1):
print(“getting value….”)
raw_sensor=mybolt.analogRead(‘A0’)
json_sensor=json.loads(raw_sensor)
if(json_sensor[‘success’]==1):
sensor_value=int(json_sensor[‘value’])
print(“sensor value: “,sensor_value)
if(len(sensor_data)>=data_count):
if(not(min_val<sensor_value<max_val)):
if(min_val>=sensor_value):
if(last_status!=1):
print(“sending message-1”)
mybolt.digitalWrite(LED,’HIGH’)
response=sms.send_sms(“Due to sudden darkness, Light is switched on!!”)
print(response)
last_status=1
else:
if(last_status!=0):
print(“sending message-2”)
mybolt.digitalWrite(LED,’LOW’)
response=sms.send_sms(“Due to sudden brightness, Light is turned off!!”)
print(response)
last_status=0
sensor_data=sensor_data[-data_count:]
avg=sum(sensor_data)/data_count
variance=0
for i in sensor_data:
variance+=(i-avg)**2
zn=int(factor*(variance/data_count)**0.5)
min_val=max(0,sensor_data[-1]-zn)
max_val=min(1024,sensor_data[-1]+zn)
sensor_data.append(sensor_value)
else:
print(“Error :”+json_sensor[‘value’])
time.sleep(10)
Thus, we have successfully implemented the ‘Anomaly Detection’ using Z-score technique, which detects any abnormality in the brightness and acts accordingly.
You can have a better view of the output in our youtube video – Z-score technique for anomaly detection
CONCLUSION:
In recent times, Anomaly detection is playing a vital role in shielding sensitive information from hackers. It is also used to examine regular usage and personalise the user experience. In this article, we have learnt what is an Anomaly, What are the types of techniques available to detect anomalies, and explanation of the Z-score technique with an example.
Written by: Batta Pruthvi
If you are Interested In Machine Learning You Can Check Machine Learning Internship Program
Also Check Other Technical And Non Technical Internship Programs
