#classificaiton

Day 137. Angles, with contours and AI classification.

by [Music] for the rest of them we need two more loops okay so the first loop is looping through each key or looping through the key list all right so remember once we hit with we're doing momentum then we need to loop through The Associated keys in the momentum so for J in key list I okay because we're looping through the ice we're good I know it's kind of complicated we're looping through the the ice element of key list which is just one of these upper level keys like momentum key used to cast a key okay and then once we're in here we want to loop through the columns of that dictionary according to that key that we're on alright because most of each key for each period we're going to calculate multiple columns of data so now we need to loop through those columns so we'll do that by doing this for K in this list dick list I J alright so what are we doing here so so dickless IJ so I is is the actual element so one of these or stochastic or Williams and then J is going to be some key inside that one so the keys according to sprints as if we're on Williams this is basically to get a give us all of the columns inside the Williams dictionary corresponding to the Jade key all right I know that's kind of complicated but just look at it for a while stare at it and you'll understand and so now we're going to do is create the column ID for this element and so the column ID is going to be the column feature plus string of J okay so that's going to be a number according and J is going to be whichever key we're on we're looking at and then plus K and K is the name of the column so for instance when we go inside the Bolinger dictionary we're going to have the name bowl or Bolinger and then 15 which is the period that we're doing it for and then k now K can be either upper mid or lower since we have three bowling or bands for each period so it's so this is going to create three different column IDs and then what we're going to do is going to set master frame column ID is equal to Dick's List and now I J okay okay so that's how it's indexed and this took me a long time to develop how to figure out how to do this well that is it now at the end of all this looping nonsense we should have a completely populated master frame now that we have the master frame we need to address a couple issues and one of the issues is that we will have man values in our data frame and for machine learning we do not want these man values so one thing that I'm going to introduce is something called the threshold okay so the threshold is is going to be I'm going to say 70% of the length of the data frame so let's do that really quick so we'll do round and we'll do point seven four seventy percent times the length of the master frame so this is going to be a number according and it's going to be a rounded number that is approximately 70% of the length of the data frame and so what we're going to do later is we're going to say if a column does not have threshold number of actual value number values that we're going to just get rid of that column completely so basically we're setting the threshold saying if we don't have this amount of data that's clean then we're going to get rid of this so next I'm just going to go ahead and rename I'm going to stress master frame to also have the price data in it so we're going to set some new columns by doing this we'll go up here open high low close ass volume lottie dottie da set that right here and actually we don't need to ask why and so just get rid of that guy and then we'll say that one is equal to prices of the same columns oh sorry I am coming down right here so right here like that okay alrighty and then another issue that we need to address I can oh she is resampled so it has this means that it's going to have empty data data in in between so what that means is that for hiking ah she you know remember we resampled it to 15 hours so that means that for our it's going to have it's going to line up with our other data but for 14 hours in between each hike and asha candle it's going to be an and value so what we want to do is we want to just back fill all those nine values so what we're going to do is master frame and we're going to get that that column that we named through this loop which we'll be hiking 15 open that is going to be equal to master frame hiked in 15 open dot still MA and we're going to use the back fill method okay and we're going to do this for all the other I cannot she high open low close so all right now that we have all of those we are going to go ahead and we are going to drop all the columns that do not meet this condition here so drop columns that have 30% or more man data okay because 30% is going to the opposite of 51st or 70% so the way we're going to do that is we're going to call it master frame clean master frame cleaned is equal to that way we have two different master frames and we can just say we're going to create it master frame copy that way we have a copy just in case something goes wrong and we don't lose any anything so now what we'll do is we'll say master frame cleans is equal to master frame cleaned dot drop not and we'll say axis is 1 so this is a column axis and we'll say Thresh Thresh and this is it is equal to threshold okay so that will drop the columns that do not meet this threshold requirement which is what we want and then we will drop our there's going to be man data for like the first so however many rows of data at the top and at the bottom so we want to drop those off as well so master frame clean is equal to mass frame clean dot drop now axis is equal to 0 so that's going to drop off data that we don't care for all right and now that that is all completed let us write it to a CSV file so that we can do it we can have access to it later later on so we'll do is we'll say master frame cleaned dot to CSV and we will put it in the data folder and we'll call it master brain dot CSV all right and let's just say complete completed feature calculations alright so if all goes well I'm going to go ahead and run it here let's say a prayer to the rain gods that it works okay looks like we just had a simple error here oh okay so this is what I did wrong here up here I said data calms what I'm meant to do is say data is equal to data indexed by this so that we're effectively dropping off the date column that was a small error hopefully there's no more as yet another small error that I found here and this one's actually really dumb so in this loop here you'll see here it says can only concatenate lists not string to list and that's because in here I accidentally set the whole like the whole string or the whole list of column features we really want is column feature I okay so hopefully that was the only dumb air I mean that's too now so to dumb errors and counting let's run it again guys well it just finished but real quick and we'll let you know that there was another error and yeah so that brings our air counter to three which is par so we're still on track here so I'm going to tell you where that error was and it had to do with the index of the hike annachi candle beta frame and that's because the feature functions when we are when we re sample the data which is right here we created this new column called symbol and so that that basically created it when we grouped it it made a multi index data frame which we didn't want and if you remember in a many videos ago when I created the item aashiq handle data frame I put a line in there but I said we won't need this until later and that line is this one right here so we originally had put it in comp it out and it was this drop level okay because that's what we want to do we want to drop the zeroth level which is the symbol index level so after you add this line or uncommented or whatever you need to do and this is in the hike annachi function file then it should work completely properly so as you can see here for my output it went through and it completed all the calculations so I'll just open it up to show you guys what it looks like and you see here the file exceeds the limit but you know it's a huge data frame that we just created so as you can see here this is the result of that loop that we made where it creates the column names momentum eight close so the 8th period momentum closed nine period momentum closed and so on so if you go through all these columns I believe there's like 73 columns I think or sixties I can't remember how many but yeah there's a lot of data that we just generated all right so this is what our feature frame looks like and it is a lot of data to handle and you'll notice if you run this by yourself that the feature collector takes a long time to gather all these features and if the number of data points goes up and we do like two or three years of data or even more then we end up spending like 30 or 40 minutes just to collect all these features so in the next video what I'm going to do is show you guys how to leverage the multi processing library so we can create a pool of processes that communicate with each other and so that we can access multiple compute computational cores and complete the process in like a fifth of the time so I hope you guys enjoyed this video in the next video or maybe in a couple videos down the road we will do that in the next video I will show you guys how to back test and to basically simulate trading with these features already and you guys have any questions or comments go ahead and ask me below you also let me know how many errors you get maybe you can feed my three errors for this file which is pretty good so I hope you guys are having a good day and I'll see you next time

[Music] alright guys welcome to part eight so in the last video we just did creating finishing creating all of the financial feature functions in this video we're actually going to create the feature space so we're going to go ahead and run through all of those feature functions that we created and populate a a data frame full of feature data for machine learning which we'll do in a couple videos from now alright so getting started just a recap we created all these feature functions and if you remember I created each function so that it could take a list of periods as an input so if you see this durations column hour duration column here and here this is the key that we're going to pass for each for each feature function so this is the list of keys that we're going to do for each one and you'll notice that for like the MACD where's all right here we have 15 and 30 but two MACD duration keys is only going to create one column of feature data so there are a couple special special cases but I'll show you guys how I'll handle that so let's get right into it the first thing that you should go ahead and do is create a new file a new PI file and on this one is I call a feature collector because what we're going to go do is go through each through each feature and populate a new data frame full of all the features and real quick before I get started I'm in the folder here that I've been working in this is like the root directory and I'm just going to create a directory called data I'm going to move our price data over into there just because we're going to start to fill this file up with like a bunch of stuff and this will just make it a little easier alright so first off let's do our imports alright so they're our imports right there you'll see I only did pandas numpy and the feature functions file which is here so we have to access to all those feature functions now inside this file so the first thing I'm going to do after that is get our data and this is just standard loading that data that we just moved into the data directory and now what I'm going to do is I'm going to rename all of the columns so that they all have names that will work in correspondence with how I built all the feature functions okay so each of the feature functions will take data that is formatted like this okay and then this line right here is just going to set the index of the data frame to be the my years pandas to daytime so it's going to change all of the data in the dot date column to data frame I mean to add a time index and it's going to set that as the index there and then we're going to go ahead and get rid of the date column so to do that I'm just going to go like this and delete date because we don't need to date columns and then I will do this prices data drop duplicates because we want to get rid of all the duplicate data before the downtime in the market because that will throw off our algorithm all right so next thing I'm going to do is I'm going to create all of those all of these keys okay and I'm going to create them as list lists of numbers as input to our functions already so you'll see that I've created a list that corresponds to each of the periods of data that we're interested in so you'll notice a couple of them only have one alright and that's because well for hiking Oishi we actually don't need to input this but we're going to use it later as a way of naming the column that we flights this data in so we're going to use all these keys for two purposes one to get our data and two to name the columns accordingly so if we name every column a specific way that we can look at all the columns and know exactly what it's referring to so now what I'll do is I'll create a list of these lists all right and we'll call that key list our key list like that let's do that and we're just going to create a list of all these keys all right so now we have a list of all these keys and so after that what I'm going to do is I'm going to go ahead and get all the features so calculate all of the features and so what I'm going to do is I'll just show you the first one let's do momentum dict so they're usually all going to be stored in in dictionaries so I'm just going to call each of them their name and then dick and then I'm going to call the function here and we're going to throw prices in there because that's our our data after we drop the duplicates which is what we want and then the according the key that goes with it so this is the periods that we're going to do it for and then what I'm going to do is just I'm just going to print one so that when it's running because it's going to take quite a while to calculate all these so that I can see like where it is and that's in the process if we get any errors we'll know where we are so I'm just going to go through and do that for all the other ones now okay you'll notice right here that for the bowling or function we actually take three inputs and that's the prices the keys the amount of periods we're doing it for and then be a number of deviations okay so don't forget to put the deviations in there for the bowling urn alright and now for the hike annachi after bowling we're going to do hike annachi but for hiking Oishi you remember that first we need to resample our data that's why we made that ohlc resampling function Oh H theory sample so we made this function and you'll notice here that we need to create a column called symbol and that's just specific for the the ability to resample the data so that's what we're going to do first is we're going to create a new data frame specifically for the hike annachi input so we're going to call it h HK a prices or name it or as I'm going to do prices not copy to get a new copy of crisis and then we're going to do we're going to create a new column called sim symbol fret and it doesn't matter what you name this so you can just name it sim or you can name it you know just name it whatever you want doesn't matter and then we'll say H ka is equal to OHL theory sample and then we'll do H ka prices and now the time periods for the time period the time frame needs to be in this format 15 hours like that okay and then we have the default column set to ask so we will mess that and so that is the end of that that's the dead frames that we're going to pass into to get our hike annachi candles now so we'll call it hiking stick is equal to I cannot see and H K a are the prices we're going to pass in and the periods will do hiking Aashiqui and print 10 so these three steps or four steps here is how we get the hike annachi feature okay so now we have at this point in the in the file we have collected all of the dictionaries that will contain the data for the periods that we pass in for each function so now what I'm going to do is I'm going to create a list of these dictionaries these resulting dictionaries because what we're going to do is we're going to loop through the list of dictionaries and the list of keys and we're going to create the column names for our new data frame with all of the features let's go ahead and do that so we call this one let's say create list of dictionaries and we'll do dickless stick list alright momentum Dix and we're going to want to get the attribute according to so each of these is going to output it's going to these are going to be the results class that gets sent back out of each function so we're going to want to access the actual dictionary inside of each one of those which we can do by this and if you go through each function you can see that that's how they're stored so you see right here that open and close our dictionaries and the scissor store open and dock close so I'm going to get all of them according to their corresponding dictionary name inside of the results class all right so that is our list of dictionaries right away you'll see that some of them are different like some of them are named clothes but there's other ones that like Bolinger is named fans the iconic candle this is named candles that price averages its average and so on and so forth so I'm not going to explain all those that you can look through the feature functions file to see why each of those is named as they are and now I'm going to create a list of so-called base column names so these are going to be the names of the features themselves okay so we're just called this column feature I guess and it's going to be the base name of each of these so momentum sarcastic okay so you'll one thing you'll notice is that for each of these lists so the dick list the key list and the column feature list all of them are in the same order and that is very important to make sure that you keep them in the same order otherwise just it's going to be a waste all right so now that we have a list of base column names we're going to go ahead and loop through all them and populate what I like to call the master frame so let's first create the master frame populate the master frame so let's call it master frame okay master faster master frame so master frame and we will do pandas dataframe with the same index is prices dot index okay so this is going to be an empty data frame with the index of the same index of prices from above okay all right so now I'm just going to talk about what we're about to do so first we're going to loop through our list of dictionaries our features so we're going to loop through this first okay so let's create an outer loop loop where we do that so for I in range 0 through the length of Dix lists okay alright so now that we're in here first I'm going to I'm going to talk about a special case and the special case is the MACD so with the MACD and right now we're creating the names of each column in the master frame that's what the purpose of this loop is to do and to populate the master frame according to each feature so what we need is to understand that for MACD we are passing in two keys here and the two keys are only going to create in a one column of data and that's very different from most of the other ones where we each key is going to create a completely different set maybe two or three columns of data for each key but MACD the two keys is only going to produce one column so I'm going to address that with the if statement so if column feature I is equal to MACD then what we're going to do is we're going to create something called the column ID column ID is going to be the column feature I okay and that's one of these and it's a string value I'm going to add let me create a string from the key list and now we're going to index the two keys according to them a CD so we know from key list that MACD is 1 2 or 0 1 2 3 4 5 6 so it's the 6th element and I'm going to access the zeroeth and the first term of that sixth element in the list so we access the 6 and then the 0 and then we're going to do the same thing for the other one key list 6 0 okay so this is going to create this is going to say MACD 15 30 is what it's going to say ok and then we'll say master frame dot column ID or doc or index by column ID so we're setting a new column in the master frame and we'll name this or will populate it with stick list I so that is what we're doing for MACD once we hit MACD and that is the only special case alright guys so I decided to cut the video here because it took much longer than expected to film all of it so in the next video I will go ahead and just pick up exactly where I left off and we'll finish up populating this feature space