AI(Practical)

  • Here we will perform different AI operations using Practical examples on different data sets.
  • In AI Execution is a strategy and development is a solution
  • AI coding is more into concepts rather than flow.
Deep Learning
  • Deep Learning is going to dominate the market next 5-7 years.
  • We can back-trace from Deep Learning to Machine Learning and advanced versions like
    • Reinforcement Learning
    • Sequence Networks
Reduce over fitting
  • Get more data
  • Expanding the data
  • Regularization
  • Dropout
  • Simplify Architecture
  • Others
Gradient Descent
  • Cost Function
  • Learning Rate
  • Minimum Batch Size
Code Envirnoment
  • Jupyter is well known environment besides many other.
    • shift+enter goes to next shell.
  • We will be using anaconda in which we have activated tensor-flow environment.
    • We launch Jupyter under tensor-flow.
API's given by Multiple companies
  • tensorflow - google
    • popular
    • takes multiple machine learning python code and integrates them into libraries.
      • We can code at tensor-flow level.
    • keras is a super api by google
    • Integrates all the tensorflow libraries and makes implementation hidden from developers.
  • pytorch - facebook
  • mxnet - amazon
    • very fast in terms of process and speed
Various Datasets
  • Ox-word flower dataser
  • MNIST dataset
  • IMDB movie review dataset
  • https://quickdraw.withgoogle.com/data
Steps to follow while building a Shallow Network
  • Build Use Case
  • Whenever we start a network we need to seed the network.
    • This is used to overcome too many loose sense in Artificial Network.
    • If there are too many looses then we cannot truly manage the network.
    • So we provide it a randomized seeding point so that it randomizes at starting point.
  • Import Libraries
  • Data
    • Import/Load this Data-set
  • Preprocess Data
    • Data Cleanup
    • Datatype change etc
  • Data Identification
  • Data Loading
    • Model Architecture Definition
      • Build the model
        • Define the model type
        • Define Layers
        • Add layers
          • AI has multiple layers and for all these layers we have multiple algorithms.
          • These algorithms have been combined into frameworks like tensor-flow which again have been combined into API's like Keras.
        • Add Output Layer
      • Compile the model
        • Standard Compilation
    • Training
      • Train the model
        • Done using fit command
          • We fit the model
          • Check accuracy.
    • Optimization
      • Fine tune the model
    Role of an AI Developer
    • As an AI developer we use these model save() and reload() functions.
    • We then provide the access to these models to python application developers who the call these using API from their code.
    • Suppose we want to predict image as a Cup or Glass they will use the function "perdict()" to get the respective output for an input.
    • Flask is a renowned framework for this.
    • Others include Microsoft Auto AML etc.
    Need of understanding of RAW Code
    1. We need to know the basics or Raw Code for 2 reasons 
      1. We need to use the correct layer for LSTM,Stacked LSTM,Bidirectional LSTM,Simple RNN or Simple Dense under different architecture and programs.
      2. When we go deeper into coding we need to be more granule about our approach.We may need to fine-tune and control high level api's like keras.
        1. Suppose we are using ADAM to fine-tune but ADAM has certain parameters Beta2 function,Beta 1 Function etc which are basic code which we may need to understand.
    Learning Approach
    • Basic Maturity
      • sample data-sets,basic codes,simple proven use-cases
    • Intermediate Maturity
      • Custom use cases,sample data representing client data,actual client data.
      • Fields like Infra-operations,Security Operations,Devops,ADM,Application Production Support,Business.
    • Advanced Maturity
      • Information Processing,differentiation,able to compete data with competition.
      • Fields like Firewall log analysis,anomaly detection,finance,lending
    • Next Gen Maturity
      • AI research paper review,research implementation, our own research,our own core research.
    Optimizing Neural Networks
    • Hyper parameter optimization
      • Initialization,Cost Function,Regularization,Dropout,Activation,Learning Rate etc
    • Model Structure Optimization
      • Number and types of Layers,No of Neurons etc
    • Data Set optimization
      • Acquisition, Pre-processing,Generalization etc.
    • Hardware optimization
      • GPU,CPU,RAM etc.
    • Tool set optimization
      • Tensorflow, MxNet, Pytorch, Keras etc.
    End to End AI development
    1. Define Use Case
    2. Understand Source Data
    3. Classify a Sample data which represents source data
    4. Classify pre-processing routines
    5. Select suitable model
    6. Start Coding
    7. Extend model as a web service.
    8. Calculate AI impact using AI driven GB.
    Real Life Use cases
    • NLP for firewall Logs
      • Firewall Log Analysis
      • Customer focus on high risk AI selected log.  
    • Prediction for Anomaly detection
      • AI enabled anomaly management.
      • Deep Learning enabled anomaly prediction.
    • Vulnerability in Applications
    • Using handwritten email process a form.
    • How to resolve the Vanishing Gradient Problem?
    • Difference between Conv1D, Conv2D, Conv3D and merging them in code.
    • Finding source and destination ip address in log data.
    • When a cyber threat happens which endpoints are likely to be attacked by hacker.
    We are using standard use-cases here but in real life we need to concentrate on Application in real life use cases.These codes are python files which can be imported to jupyter. But it is preferred to copy and execute these line by line for better understanding.

    Lets start our First Example on MNIST handwritten digit database.
    Example 1
    • Classify a hand written digit.Postal service gets letters and everyone writes zip codes with hands.There is a need for a computer algorithm which could read zip-codes and classify them on basis of numbers.
    • Wanted to achieve basic elementary Computer Vision.
    • 60,000 digits handwritten by various people were collected.
      • This became a data set called as MNIST data set
      • Modified National Institute for standards and Technology.
    • Numpy library is used for numerical processing.
    • Random seed setting.It provides randomization of neural network.
      • An uncontrolled randomized neural network may cause issues on adhoc basis which may be difficult to fine tune.
      • We put a random seed to control this.
      • We are using 42 here.
      • It helps to stabilize network from a randomized perspective.
    • Keras has multiple sub modules such as
      • Dataset Modules
      •  Models
      • Layers optimizers etc in Keras.
    • We are importing MNIST here.
      • In MNIST load data 
        • (X_train,y_train),(X_test,y_test)=mnist.load_data().
        • X_train,X_test has training and testing images.
        • y_train,Y_test has data related to these images i.e. it is the classifier.
    • We will import sequential/feed forward type model here.
    • Import dense layer type.
    • Then we define the optimizers for the loss function
      • For example if out of 10 images 6 are predicted rightly then we will have 60% accuracy.
      • The model will relearn again.
      • This learning process is defined by multiple functions for example adam, adadelta, adagram and so on.
        • We had Gradient descent and Sarcastic Gradient Descent,now we have optimized engines.
        • In real life we only use optimizer engines
        • For this example we will use Sarcastic Gradient Descent.
    • Now we will load data which are handwritten image files structured into 28 X 28 pixels.
      • Load data command is used to load the data.
      • For every image data there is a label and a classifier.
        • If Image is of no. 5 then data represents binary form of image and classifier represents number 5.
    • Next we have testing data
      • This is used to overcome problem of over-fitting i.e. model should not struggle when a new data comes in.So we divide data into 2 buckets training data and testing data.
      • Here Keras has already done it but we may need to do it manually which may be in split of 60:40,7-:30 or 80:20 depending upon the data.
     # coding: utf-8  
 # # Set Seed for reproducability  
 # In[1]:  
 import numpy as np  
 np.random.seed(42)  
 # # Load dependencies  
 # In[2]:  
 import keras  
 from keras.datasets import mnist  
 from keras.models import Sequential  
 from keras.layers import Dense  
 from keras.optimizers import SGD  
 # # Load Data  
 # In[3]:  
 (x_train,y_train),(x_test,y_test) = mnist.load_data() #We are loading data  
 # In[4]:  
 x_train.shape #check how data structure is  
 # In[5]:  
 x_train[0]  
 # In[6]:  
 # 6000 samples for training data.  
 y_train.shape  
 # In[7]:  
 y_train[0:99]  
 # In[8]:  
 # 10000 samples for testing data.  
 x_test.shape  
 # In[9]:  
 #Representation of the number in data set   
 #To check what number is run y_test[0] which is a classification of number  
 x_test[0]  
 # In[10]:  
 # 10000 samples for testing data  
 y_test.shape  
 # In[11]:  
 # We find that above representation was of number 7  
 y_test[0]  
 # # Preprocess Data  
 # In[12]:  
 # We are making 28 by 28 images into single vector of 784 and change type to float 32.  
 # Since Y is a single number only a classifier we will not preprocess the Y.  
 x_train= x_train.reshape(60000, 784).astype('float32')  
 # In[13]:  
 x_test= x_test.reshape(10000, 784).astype('float32')  
 # In[14]:  
 x_test[0]  
 # In[15]:  
 x_train /= 255  
 x_test /= 255  
 # In[16]:  
 x_test[0]  
 # In[17]:  
 # We have a dta set of 10 numbers between 0-10  
 # Out put is a probability i.e it may say 100% its 5 or 60% its 5 and 40% looks like 9  
 # So we convert our Y data into an array of 10 digits  
 # This process is applicable when we have a non binary classification  
 n_classes=10  
 y_train=keras.utils.to_categorical(y_train, n_classes)  
 y_test=keras.utils.to_categorical(y_test,n_classes)  
 # In[18]:  
 # Now we see that it gives us an array result and we have 1 at 7th Position  
 # So the number is 7  
 y_test[0]  
 # # Design neural network architecture  
 # In[31]:  
 #We are building a sequential model here  
 #And then we are adding the layers and using the concerened activation function  
 #Mostly Relu is used as activation function but here for a simple model we are using sigmoid.  
 #In the output layer we are using "softemax" since it is a probability distribution as seen above  
 model=Sequential()  
 model.add(Dense(64, activation='sigmoid', input_shape=(784,)))  
 model.add(Dense(10, activation='softmax'))  
 # model.summary()  
 # # Configure Model  
 # In[20]:  
 #We are using mean_squared_model to figure out deviation  
 #The actual value can be less than the target or more than the target to nullify that we are using squared model.  
 #For advanced models we use categorical cross entropy  
 model.compile(loss="mean_squared_error",optimizer=SGD(lr=0.01), metrics=['accuracy'])  
 # # Train  
 # In[21]:  
 #We train model on random samples of batches which is called as batch size  
 model.fit(x_train,y_train,batch_size=128,epochs=50,verbose=1,validation_data=(x_test,y_test))  
 # In[22]:  
 # Here accuracy is 54.98 % which is not that great  
 # In[29]:  
 #displaying images can be done as follows  
 # We are again reshaping data into 28X28 bitmap   
 import matplotlib.pyplot as plt  
 todisplay=x_train[0].reshape((28,28))  
 plt.imshow(todisplay, cmap='gray')  
 plt.show()
    So we see that when we run the model we have 54.98 % accuracy in this model which is not reliable.
    Example 2:
    • We will build a shallow network here.
    • A shallow network has a model with 1 input,1 output and 1 hidden layer. 
    1:  # coding: utf-8  
2:  # In[1]:  
3:  #We are going to build a shallow Network Here and then convert it into Deep Network  
4:  #Importing Libraries  
5:  import numpy as np  
6:  np.random.seed(42)  
7:  # In[2]:  
8:  import keras  
9:  #import Dataset  
10:  from keras.datasets import mnist  
11:  #import Type of Model  
12:  from keras.models import Sequential  
13:  #import Type of Layer  
14:  from keras.layers import Dense  
15:  #import type of optimizer to reduce cost  
16:  from keras.optimizers import SGD  
17:  # In[3]:  
18:  #x_train is the actual image and y_train is a classifier for the image  
19:  (x_train,y_train),(x_test,y_test)=mnist.load_data()  
20:  # In[4]:  
21:  x_train.shape  
22:  # In[5]:  
23:  y_train.shape  
24:  # In[6]:  
25:  x_test.shape  
26:  # In[7]:  
27:  y_test[0]  
28:  # In[8]:  
29:  #Preprocessing Data  
30:  x_train=x_train.reshape(60000,784).astype('float32')  
31:  x_test=x_test.reshape(10000,784).astype('float32')  
32:  # In[9]:  
33:  x_train[0]  
34:  # In[10]:  
35:  x_train /= 255  
36:  x_test /= 255  
37:  # In[11]:  
38:  x_train[0]  
39:  # In[12]:  
40:  n_classes=10  
41:  y_train=keras.utils.to_categorical(y_train, n_classes)  
42:  y_test=keras.utils.to_categorical(y_test, n_classes)  
43:  # In[14]:  
44:  y_test[0]  
45:  # In[15]:  
46:  model=Sequential();  
47:  model.add(Dense(64,activation = 'sigmoid',input_shape=(784,)))  
48:  model.add(Dense(10,activation = 'softmax'))  
49:  # In[16]:  
50:  model.summary()  
51:  # In[17]:  
52:  model.compile(loss='mean_squared_error',optimizer=SGD(lr=0.01),metrics=['accuracy'])  
53:  # In[18]:  
54:  model.fit(x_train,y_train,batch_size=128,epochs=50,verbose=1,validation_data=(x_test,y_test))

    Example 3
    • We are building a deep neural network here.
    1:  # coding: utf-8  
2:  # In[1]:  
3:  import numpy as np  
4:  np.random.seed(42)  
5:  # In[5]:  
6:  import keras  
7:  from keras.datasets import mnist  
8:  from keras.models import Sequential  
9:  from keras.layers import Dense  
10:  from keras.layers import Dropout  
11:  from keras.layers.normalization import BatchNormalization  
12:  from keras import regularizers  
13:  from keras.optimizers import SGD  
14:  # In[6]:  
15:  (x_train,y_train),(x_test,y_test)=mnist.load_data()  
16:  # In[7]:  
17:  x_train=x_train.reshape(60000,784).astype('float32')  
18:  x_test=x_test.reshape(10000,784).astype('float32')  
19:  # In[8]:  
20:  x_train /= 255  
21:  x_test /= 255  
22:  # In[9]:  
23:  n_classes=10  
24:  y_train=keras.utils.to_categorical(y_train, n_classes)  
25:  y_test=keras.utils.to_categorical(y_test, n_classes)  
26:  # In[11]:  
27:  model=Sequential()  
28:  model.add(Dense(64,activation='relu',input_shape=(784,)))  
29:  model.add(BatchNormalization())  
30:  model.add(Dropout(0.5))  
31:  model.add(Dense(64,activation='relu'))  
32:  model.add(BatchNormalization())  
33:  model.add(Dropout(0.5))  
34:  #model.add(Dense(64,activation='relu'))  
35:  #model.add(BatchNormalization())  
36:  #model.add(Dropout(0.5))  
37:  model.add(Dense(10,activation='softmax'))  
38:  # In[12]:  
39:  model.summary()  
40:  # In[13]:  
41:  model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])  
42:  # In[ ]:  
43:  model.fit(x_train,y_train,batch_size=128,epochs=200,verbose=1,validation_data=(x_test,y_test))

    Example 4
    • Natural Language Processing Example
    • We are performing a sentiment classification analysis regarding a movie using supervised learning mechanism
      • We have 25000 data-sets in imdb with supported data classifiers along.
      • Based on the classifiers our algorithm will learn the new sentiments whether they are positive or negative.
    • NLP's are represented as Embedding Word Vector.
     # coding: utf-8  
 # In[1]:  
 import keras  
 # We will be classifying a sentiment as positive or negative on basis of this data  
 from keras.datasets import imdb  
 # This helps us in getting structured data so that we can get the relevent amount of data we want.  
 # Some people might have given 50 words review some 100 words with same sentiment.  
 # We will pad our data in this case  
 from keras.preprocessing.sequence import pad_sequences   
 from keras.models import Sequential  
 # When the input comes in based on the dimensions we can apply Flatten to make the model a bit linear and easier.  
 from keras.layers import Dense, Flatten,Dropout  
 from keras.layers import Embedding  
 # In[2]:  
 # Always use separate section for hyper parameters.  
 epochs = 4  
 batch_size = 128  
 n_dim=64  
 n_unique_words=5000  
 n_words_to_skip=50  
 max_review_length=100  
 pad_type = trunc_type = 'pre'  
 # Architecture Parameters  
 n_dense=64  
 # Dropout Ratio is 50%  
 dropout=0.5  
 # In[3]:  
 #Loading Data  
 (x_train,y_train),(x_valid,y_valid)= imdb.load_data(num_words=n_unique_words, skip_top = n_words_to_skip)  
 # In[4]:  
 x_train[0:6]  
 # In[5]:  
 for x in x_train[0:6]:  
   print (len(x))  
 # In[6]:  
 y_train[0:6]  
 # In[7]:  
 len(x_train)  
 # In[8]:  
 len(x_valid)  
 # In[10]:  
 x_train=pad_sequences(x_train, maxlen=max_review_length,padding=pad_type,truncating=trunc_type, value=0)  
 # In[12]:  
 x_valid = pad_sequences(x_train, maxlen=max_review_length,padding=pad_type,truncating=trunc_type,value=0)  
 # In[13]:  
 x_train[0:6]  
 # In[14]:  
 for x in x_train[0:6]:  
   print (len(x))  
 # In[18]:  
 # Build The model  
 model=Sequential()  
 model.add(Embedding(n_unique_words, n_dim, input_length=max_review_length))  
 model.add(Flatten())  
 model.add(Dense(n_dense, activation='relu'))  
 model.add(Dropout(dropout))  
 model.add(Dense(1,activation='sigmoid'))  
 # In[19]:  
 model.summary()  
 # In[20]:  
 model.compile(loss='binary_crossentropy', optimizer='adam',metrics=['accuracy'])  
 # In[24]:  
 model.fit(x_train,y_train,batch_size=batch_size,epochs=epochs,verbose=1,validation_data=(x_valid,y_valid))

    Example 5
    • RNN Example
     # coding: utf-8  
 # In[3]:  
 import keras  
 from keras.datasets import imdb  
 from keras.preprocessing.sequence import pad_sequences  
 from keras.models import Sequential  
 from keras.layers import Dense,Dropout,Embedding, SpatialDropout1D  
 from keras.layers import SimpleRNN  
 # In[13]:  
 #Docomenting Hype Parameters  
 epochs=5  
 batch_size=128  
 n_dim=64  
 n_unique_words=10000  
 max_review_length = 100  
 pad_type='pre'  
 trunc_type='pre'  
 drop_embed=0.2  
 n_rnn=256  
 drop_rnn=0.2  
 # In[14]:  
 (x_train,y_train),(x_valid, y_valid) = imdb.load_data(num_words=n_unique_words)  
 # In[15]:  
 x_train = pad_sequences(x_train, maxlen=max_review_length,padding = pad_type, truncating=trunc_type,value=0)  
 x_valid = pad_sequences(x_valid, maxlen=max_review_length,padding = pad_type, truncating=trunc_type,value=0)  
 # In[16]:  
 #Building the model  
 model=Sequential()  
 model.add(Embedding(n_unique_words,n_dim,input_length=max_review_length))  
 model.add(SpatialDropout1D(drop_embed))  
 model.add(SimpleRNN(n_rnn, dropout=drop_rnn))  
 model.add(Dense(1, activation='sigmoid'))  
 # In[17]:  
 model.summary()  
 # In[18]:  
 model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])  
 # In[20]:  
 model.fit(x_train,y_train, batch_size=batch_size,epochs=epochs,verbose=1,validation_data=(x_valid,y_valid))

    Example 6
    • LSTM Example
     # coding: utf-8  
 # In[1]:  
 import keras  
 from keras.datasets import imdb  
 from keras.preprocessing.sequence import pad_sequences  
 from keras.models import Sequential  
 from keras.layers import Dense,Dropout,Embedding, SpatialDropout1D  
 from keras.layers import LSTM  
 # In[7]:  
 #Docomenting Hype Parameters  
 epochs=5  
 batch_size=128  
 n_dim=64  
 n_unique_words=10000  
 max_review_length = 100  
 pad_type='pre'  
 trunc_type='pre'  
 drop_embed=0.2  
 n_lstm=256  
 drop_lstm=0.2  
 # In[8]:  
 (x_train,y_train),(x_valid, y_valid) = imdb.load_data(num_words=n_unique_words)  
 # In[9]:  
 x_train = pad_sequences(x_train, maxlen=max_review_length,padding = pad_type, truncating=trunc_type,value=0)  
 x_valid = pad_sequences(x_valid, maxlen=max_review_length,padding = pad_type, truncating=trunc_type,value=0)  
 # In[10]:  
 #Building the model  
 model=Sequential()  
 model.add(Embedding(n_unique_words,n_dim,input_length=max_review_length))  
 model.add(SpatialDropout1D(drop_embed))  
 model.add(LSTM(n_lstm, dropout=drop_lstm))  
 model.add(Dense(1, activation='sigmoid'))  
 # In[11]:  
 model.summary()  
 # In[12]:  
 model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])  
 # In[13]:  
 model.fit(x_train,y_train, batch_size=batch_size,epochs=epochs,verbose=1,validation_data=(x_valid,y_valid))

    Example 7
    • Convolutional Neural Network
    • We have built capability for Computer Vision.
     # coding: utf-8  
 # In[1]:  
 import numpy as np  
 np.random.seed(42)  
 # In[2]:  
 import keras  
 from keras.datasets import mnist  
 from keras.models import Sequential  
 from keras.layers import Dense, Dropout  
 # Since mnist is a 2D image we are using Conv2D  
 # MaxPooling2D is used to extract the features further it has made our computation focused,efficient and relevent  
 # Maxpooling is used along with CNN  
 from keras.layers import Flatten, Conv2D, MaxPooling2D  
 # In[3]:  
 (x_train,y_train),(x_test,y_test)=mnist.load_data()  
 # In[5]:  
 x_train=x_train.reshape(60000,28,28,1).astype('float32')  
 x_test=x_test.reshape(10000,28,28,1).astype('float32')  
 # In[7]:  
 x_train = x_train / 255  
 x_test = x_test / 255  
 # In[8]:  
 n_classes = 10  
 y_train = keras.utils.to_categorical(y_train, n_classes)  
 y_test = keras.utils.to_categorical(y_test, n_classes)  
 # In[11]:  
 model = Sequential()  
 # 32 is the number of neurons  
 model.add(Conv2D(32,kernel_size=(3,3), activation='relu', input_shape=(28,28,1)))  
 model.add(Conv2D(64,kernel_size=(3,3), activation='relu'))  
 model.add(MaxPooling2D(pool_size=(2,2)))  
 model.add(Dropout(0.25))  
 model.add(Flatten())  
 model.add(Dense(128,activation='relu'))  
 model.add(Dropout(0.5))  
 model.add(Dense(n_classes,activation = 'softmax'))  
 # In[12]:  
 model.summary()  
 # In[14]:  
 model.compile(loss = 'categorical_crossentropy',optimizer='adam',metrics=['accuracy'])  
 # In[16]:  
 model.fit(x_train,y_train, batch_size=128, epochs=5,verbose=1,validation_data=(x_test,y_test))



    eop

    Comments

    Post a Comment

    Popular posts from this blog