Explore Data Science Foundational Concepts

0:00Hello and welcome to the ITCA Data Science Foundations course. My name is Jonathan Barrios and I'm super excited to work with you to illuminate all the different domains that come together to form the Data Science Foundations. But first, let's talk a little bit about me and my approach to this course. I'm a Data Science and Machine Learning Instructor and I'm passionate about combining deep learning plus music. That being said, I create content with a goal of simple understanding without complicated jargon.

0:35And the goal is that we're going to be using simple to apply terms. And hands-on application with something called Google's CoLab. And this is a super fun little notebook that we can enter code and it'll just spit out the answers. And it's a great way to work with Data Science, especially when you're working with the foundational concepts. In short, I don't think that Slides Alone really is a useful delivery of information, especially Data Science concepts.

1:14Moreover, foundational concepts. And that's why we're going to use Google CoLab because the Slides Alone really don't cut it. So we're going to add stories, hands-on or use cases and hands-on applications to really cement all of these concepts. But now let's go ahead and take a look at all of the domains that make up the Data Science Foundations. And when we're talking about Data Science Foundations, this is exactly what we 're talking about. So Data Science here is really an umbrella term.

1:45So if you want to think of this as like a larger Venn diagram, so this would be number one. And then Data Structures, right, that sort of all be housed in here, all of these in different proportions, right? There's going to be a little bit more than each one. So here we have Data Structures and this is just talking about the different data types. And this could include static, trees, which is a type of data structure, linear , and nonlinear, right?

2:18So line data and non-line data. This is probably the easiest way to explain this at this stage, right? Because we're just talking about the foundational concepts. And then moving on, we'll dive into statistical analysis. And these are going to be methods and techniques that study the populations and samples. And let's say you have a sample of all of the voters and then, or excuse me, the population. So this is all of the voters. And then here is a sample.

2:48So you take a small, like let's say 10% subset. And other techniques. And the idea here is that we want to find trends and other useful pattern hidden inside of raw data. Moving on to data management, really about storing how to store data. And we're really going to dive into relational. And non-relational databases.

3:20And some SQL SQL. And don't worry if you're not familiar with relational or non-relational or even SQL, because we're going to go over each one of those. And then we'll dive into data governance. But what is data governance? And against, don't worry if you don't have a working definition of this. Just think data management. For now, that includes security and protection.

3:55And more importantly, how to work with big data and how to manage that. Because it can be tricky. And then we'll dive into data access and protections. And here you can think about managing permissions. And securing data. Which is this little bit of overlap between data management and data governance .

4:31We'll talk about that because we used security for both of these, but they're a little bit different. And more importantly, when you're working with third party vendors. And here for data discovery and collection, this can include hypothesis testing . Understanding business models. And what is the role? And what is data taxonomy?

5:05And then we move to machine learning, my favorite. We're not really going to get into machine learning per se, but we will discover the data types. And the some basic concepts. Because it's super important, right? It's all the rage right now. And statistical analysis. You may have heard machine learning and also statistical learning. They kind of go hand in hand.

5:36So we'll talk about that relationship. And then finally, we have data presentation. Probably the most important because here at the very end, you're going to explore how to summarize the entire data science process, right? Because you're going to have to communicate this, right? And that's the most important part. Communicate to stakeholders. Because once you find that information or you use data science and your understanding of data structures, right?

6:09And the different data types and you collect that and use some sort of statistical analysis, take it through the organization, using data management and data governance. And then here are the access and protections, making sure that you don't compromise that data. And then here you actually find those trends, use machine learning if you need to. And then here is where we communicate that to the stakeholders. And then you can take some sort of action and increase the value of your organization. And what we're going to do is take all of these and turn it into one journey.

6:44And we're going to go through this journey together. And that slide, or video I should say, is next. So we're going to dive right into data science and I will see you there.

0:00Welcome back. In this video we're going to dive into data science and what is data. So again in the previous video we talked about everything that we're going to learn in this data science course, the Data Science Foundations course. This is this first scale in that course and we're going to start off with answering the questions, what is data science and what is data? Oops. Alright so what is data science? Data science is the process of extracting valuable insights from raw or unprocessed data. So this is really the key.

0:41There's nothing really invaluable inside of this raw data, which is also unprocessed data, right? Data that you would see as is. And again like we said before it combines techniques from statistics, software engineering, business analysis and other fields as we discussed. And here is the takeaway, the goal. So the goal of data science is to uncover useful knowledge and information from large and complex data sets and various formats. So the formats are the data structures and that's really tricky because we have to work with all of these structures and then make them available so that we can actually get the useful knowledge out of that raw data right there. Let's take a use case for example. Let's say that we have some sort of a retail store, right? In a data scientist can analyze customer purchases. So these could be the purchases, maybe some website traffic and most importantly demographics, right? What is the demographic of the user? Then

1:56we can take all of this and with data science we can make predictions about future buying. So let's say predict. So buying behaviors, right? Because if you know what those behaviors are then you can make personalized marketing strategies. And the takeaway here is that we're processing and interpreting this data so that businesses can enhance customer satisfaction, optimize inventory because imagine if you have no inventory then you're going to make zero money.

2:39And ultimately we want to drive sales growth and demonstrate the power of data science to our stakeholders because we can transform raw data into actionable knowledge. And that's the takeaway, actionable knowledge. Because it comes from raw data right up here that it's raw or in process and really doesn't have that much value. So we're taking no value and turning it into value. Pretty cool. And those would be called insights, right? So insights are when you extract, whoops , let me go back, insights are when you extract useful knowledge and information from usually big, whoops, let me turn my pan back on that's what's happening. I 'm trying to write this, turning it into a stylus. So extract useful information, knowledge from big and complex datasets. And that's going to be also what we call big data. We're going to dive into that because that's a hold. It's a big domain unto itself. Let's say that. And there's going to be a lot of different formats that you can might be familiar with comma separated values, or maybe even a dot pi file, which is a Python file or a IPY and B, which is this is a IPython notebook or a Jupiter notebook. These are all very popular data science data formats, right? And so you have to be familiar with that. Because

3:59without that, you can't offer these insights. For example, maybe we have like a hospital, let's say that this is like, hopefully this will look like a hospital and big door here. Supposed to be like a little cross, right, like a hospital. We could say this is health care, right? Some sort of a health care organization. And what they can do is analyze patient records, medical images, maybe even genetic data to find patterns, right? And we want these patterns to lead to, I guess it could be better diagnoses and treatment, right? So ultimately you're going to get better care. And also maybe even make the operations more efficient than drive the costs down. So that's what we do. We bring value to our organizations. And we can show how important it is to manage and analyze different types of data, right? All of these data. If you don't know how to do that, you're not going to get these insights here. Right, that's the takeaway.

5:08And here we can add something really important because data science requires domain knowledge. And that's a key takeaway to answer what is data science. And here's why. We use domain knowledge to guide business decisions, right? And here's a good one, data driven marketing, because we're using data to make marketing decisions or just making decisions in general and targeting of services with a goal of improving customer and stakeholder expectation. We've talked a little bit about stakeholders and how we take all of the steps in the data science foundations to share that knowledge, right?

5:51We want to share how to increase revenue or add value to your organization using some kind of domain knowledge. For example, let's think of a retail company that uses insights from customer data to tailor marketing campaigns. And these targeted services can be more effective by doing so. So by basing decisions on solid data, businesses can make better informed choices, right? And in this case, they can maybe boost customer engagement and meet expectations better for their customers. And this approach ensures that we're taking the strategies and aligning them with real world insights. And for this case, for retail markets. And last but not least, we have strategy. And this is when we leverage the ability to identify useful patterns and then use data driven insights to avoid expensive business decisions. For example, if we're analyzing sales data, a company can spot some kind of a trend. Let's say that they're doing some stocking and they want to avoid like adding too many of one product, right? If it's not

7:07popular, right? Like let's say that they have 100 here, but there's only a demand for three, something like that. This is a big difference. And this is how we can use our resources wisely and use make strategic strategic decisions based on again, solid data. And that's why we say things like data driven. All right, so that's it for this video. And I'll see you in the next video when we dive into what is data, right? We're getting into these foundational concepts. See you there.

0:00Welcome back. In this video, we're going to answer the question, what is data? And dive into a little bit of the nuances of the different data structures so that when we get into those data structures, especially when we get to that skill, it's not going to be like, what is all this stuff, right? We have already laid the foundation and that's the goal of this video. Alright, so what is data? Data is information that can be collected, measured, manipulated, and analyzed. So we've talked about these a little bit, right? We're just making sure that we're on the same page. It can be numbers, text, images, or other information. So now we're getting a little bit more granular, right? Numbers, text, and then images . It can be sunsets, right? Or what is this other type of information? Well, there's going to be what seems like an endless amount of information or data structures , right? And that's just if you're not familiar with these data structures. After a while, you see that you work with certain kinds more often than others. But there's always going to be these more common and less common data structures.

1:10And this is a paramount skill to have because one of the most important things for data scientists is to work with data, right? With the science of data. So we're going to make sure that all of this is solid as we have this iterative journey and answer these questions along the way. But more importantly, let's answer this question. Data is some kind of information, right? It has no value because it's raw and it's self-referencing, right? So let's talk about this referencing . It's not connected to anything else, right? And that makes it stand alone. So if it 's not connected to any other context, then it's really like it doesn't, it's like not valuable, right? But it is malleable, meaning that you can change it, but it lacks integrity. So these are some really important attributes of data. And this is

2:25what is data. So I think we're good to go to the next one. Now we know the data is information that can be collected, measured and analyzed. It can be numbers, text, images or other types of information. But now we're going to get more granular and talk about the two big categories of data. Qualitative and quantitative. So what I would suggest here is to think about the root of the world, root of the word. So qualitative data, I'm hearing quality, right? So that's the root. And it's referring to the descriptive information, right? That is not numbers. Text, images, it's, let me put it this way. It's not numbers, right? But it is text, images or categories. Right? The most famous examples are we colors, names or labels, right? These are not numbers. And if it ever does use numbers, they don't have a meaning, right? They're not being, it just kind of like male is one, female is zero or the other way around, right? These numbers don't represent anything except for male females or like genders in this case. So

3:41what is quantitative data? So here, quantitative, like quantity, quantitative data is a numerical information that can be measured or counted because you can do that with numbers, right? You can count your age, how many, what is the collective age of a group of people? What is your height or the median height of a group of people? And the same thing with temperature, you can get the median temperature, the actual temperator, but it's actually some sort of numerical piece of information, like 74 degrees. I wish it was 74 degrees. Okay, so now we're circling back to what is unprocessed versus process data because we talked about this raw data, right? It has no inherent meaning or value for that matter .

4:26And data science process allows us to draw conclusions and gain insights. To be meaningful, the data has to have some sort of dimensionality, connectivity, and transformability. So dimensionality, multiple features or aspects, and I promise we'll get into that. If this is not clear, don't worry, we're gonna get into each one of these. Connectivity, the relationships between the data points , you remember when we said that the data was isolated, not connected, had no context? Well, that's why it was not very useful. So it needs to be connected and have relationships with data points. That's the beginning of the value and transformability, the ability to be cleaned and formatted so that we can perform analysis. This is a really big takeaway. Cleaning is where we spend, you would be surprised, like how much time we spend cleaning data. You would think that data science, you maybe have a data science background, you may already be a data scientist, so you may already have your own understanding of this. In short, it's counterintuitive to think that a data scientist is going to be spending most of their time cleaning data, but it's a thing. So if you didn't know that, we spend a lot of time just doing pre-processing so that we can get to the fun

5:44part here of analysis and other cool techniques that are more advanced. As promised, we're going to dive deeper into dimensionality. So here dimension ality is the number of features or variables in a data set. For example, the dimensions for time can be days, minutes, and seconds. However, in data science, techniques like dimension, dimension reduction or dimensionality reduction are used to simplify the data set. So when you want to reduce the complexity, you can reduce the dimensions by selecting important dimensions such as date and hour without seconds because the context is saying that, "Hey, we just need to know what date and what hour it was so we don't need that." So we can reduce the complexity by taking off the dimensionality of seconds in this case. So if you're looking at a data set of cars, for example, you might be thinking of a spreadsheet like this. And you might have attributes like horsepower, weight,

6:59and what else? Let's say fuel efficiency like miles per gallon, right? Let me write that. MPG and so on and so on, right? These are all the different attributes or dimensions. So the total number of features in the data set of the data set has three features, for example, horsepower, weight, fuel efficiency or miles per gallon. We can say that this data set has three dimensions. All right , so I said horsepower, weight, and miles per gallon. All right, so this is a good stopping point. Now we'll see you in the next video when we dive into an example using Google Colab. See you there.

0:00Welcome back. Before we dive into Google Colab, I really feel like I should explain what Google Colab is. If you already know what a Jupiter notebook is, locally, it's a cloud version of that that is available on the browser. You don't have to install anything. You can just start using Python right away. It's actually the best way to learn Python. It's just to use Google Colab so you don't have to spend all that time getting your development environment up like if you were like a computer scientist, right? Where data scientists, we have some computer science, but we're not exclusively going to be using the same tool. So this is a great tool. So let's get into it. Let me explain how all of this works. Okay. So here, Google recommends using Chrome. No thanks. I use Brave. It's a little bit I can protect my data more than Chrome, which is going to power the this organization. So I'm going to say don't switch. And then I'm going to say coll ab. One word, one L. And then it says welcome to collab. You can click on this or this one.

1:04It really doesn't matter. So if you click on either one of these, it's just going to let 's click on this one. I'll just show you. And then you go to new notebook or open collab . So you have to have a Google account and be logged in. And once you do that, you click on new notebook, you'll get something that looks like this. And I will give you a quick rundown. So here you can work with Drive, which is pretty amazing. And there's also a terminal, which is a newer feature. And you can also use AI, but we turned it off because we don't want to use AI just yet. It's not great yet. It needs a few more iterations before we can actually use it for data science. So here, what we can do is just say, let me prove it to you so I can use the exclamation mark and that talks to the terminal. Right. So instead of opening the terminal, I can just use this exclamation mark. And then I'm going to say here, Python dash dash version, and then shift enter, or you can push play. I mean, that was pretty fast, right ? So we have this in Python installed. You don't have to do anything. You can just start right out of the box.

2:12Boom, start working with Python. For example, let's do the print Hello world. So you see that I'm not using any exclamation marks. And I'm just going to use you can use single or double quotes. Let's use double quotes and hello. World exclamation mark shift enter or alternatively, you can hit this play button. And then here you get the output, right? So this is Python and we've proven it. This is normally how you do that. So now you know what Google Colab is, and you'll know more about how it works as we progress through this video. But more importantly , for data science and machine learning, you can actually go over here to change runtime type and boom, you get these, this isn't totally wild. You can get all these GPUs. Not all of them are going to be available for free, but you can also select high RAM. So I'm not going to do that. But right now I'm just using the CPU. So if you're not going to be needing a GPU, always use a CPU so that we save those resources. Otherwise, Google will get mad and limit you and your ability to use these. So we have a free GPU. How cool is that? All right, so then we go over here. And this is going to be available to you. And right in under the video, just click on that in this

3:28notebook will open up. And then what we're going to do here is this run through this. So if I hit shift, well, let me explain everything before I run this. So here, these are all the packages that we're going to be using. NumPy is actually pandas is built on top of NumPy. And we're going to be using pandas for data manipulation and matplot.pyplot as PLT. Also, these are just aliases so that we can write this instead of these. So it's just an abbreviation. So it's just NumPy pandas mat plotlib. And seaborn is also for visualization and scikit learn. This is something that we 're going, this is for decomposition. This is for the example, I don't want to get into this here. But everything up here is for visualization and data manipulation. And then here we're going to create some synthetic data with five dimensions. And I'm going to show you how I remove these features, right? That's the whole point. And then here, I'm going to create a data frame. And what a data frame is, in essence, is a table. So like a spreadsheet. So we'll get into that. We're also going to really break down spread sheets and open spreadsheets and get into that. But for right now, this is kind of a programmatic version of that. So let me go ahead and enter shift, enter. And then it's going

4:44to run this. The first time you do this, you can see over here is is connecting. It just takes a minute, connect to some computer in Google's warehouse. And then once we have that, boom, here are our features. So like I said, this is rows and columns, just like a table. And here are features 123 45. So if you wanted to reduce that, you can use PCA, we're going to use that for dimension reduction. So let me show you what's happening here, we're going to say that we want to define how many components we're going to remove, right? So we're going to reduce the data columns one and two. And let's just take a look at that. No need to understand what's happening here, other than what I'm showing you, that we're going from five features to two features, right? Remember, we're talking about horsepower, weight, miles per gallon, maybe cylinders and some other feature. Maybe we just want the first two. I just want to know what the mile per gallon is in the way to car. Right, then you reduce the dimensions.

5:43And that's exactly what we're talking about. But you may be thinking, well, I sort of get it, but I don't get it. Let me show you exactly what's happening here. So here we have x, y, and we have x, y, z. So this is a three dimensions, and this is two dimensions. So you can literally see that we're reducing the dimensions manually. We took the third dimension on it here. So now we have two dimensions and the three dimensions. So hopefully that makes a little bit more sense. I feel that visualizing data is really the most superpower thing that we can do as data scientists. I think that this really is the takeaway. And on that note, I'll see you in the next video on connectivity. See you

0:00Welcome back. In the last video, we talked about dimensionality and we were reducing dimensions, dimension reduction. And here we're going to continue with connectivity. Connectivity in the context of raw data, this is super important. We're going to circle back on that versus contextualized data. I guess that's the comparison that we're making, right? Raw data compared to connected data. So connectivity are the relationships and dependencies between different data points or features within a data set. So we talked about these different features.

0:44So again, connectivity are the different relationships between features. Furthermore, it illuminates how data points are linked or correlated, which can be crucial for understanding the underlying patterns and presenting valuable insights. Again, this is increasing the value to your organization. And that when you do that, you basically get a raise. You get a promotion. This is a good thing. This is what your job is really bringing value to your organization.

1:18For instance, if I said 100, what does that mean? Right? That's raw data. And but we don't even know what it means. But how about if I was talking about chance of rain? And then I added this, my boom, I've actually added some context and I connected it to a percentage and chance of rain. So again, imagine a table. And then each one of these will have a number, right? And those numbers could be 100, like we did down here or 70 or whatever.

2:03And these numbers just won't make sense until you understand that this one is going to be for temp, right? This one's going to be for humidity. And this one's going to be for the date. When you do that, then you say, Oh, this is 100% chance of rain. And we know the temperature is going to be 75 degrees, things like that. I did get into the context a little bit early when we're talking about chance of rain. Whoops, let me turn the pencil on. There we are contextualized data. So when I was talking about chance of rain, that was context, right? And so we could actually get that context from the features. But let's take a look at the definition. Contextualized data enhances raw data, right? So by adding layers of information that provide depth and context, these could be features. So beyond mere numbers and facts, it includes elements such as personal perceptions, individual biases and cultural or industry specific contexts.

3:06And the takeaway here is that what we're doing is helping us have a better understanding of the data, right? Especially in the context of real world implications and in making informed decisions based on a fuller picture. So if you didn't know that this was chance of rain and you just had a 100 boy, what are you going to think with that, right? You 're not going to really be able to provide this informed decision, right? It's like, well, it says a 100. But if it's a 100% chance of rain, then that's what we're adding here context.

3:40I'm making this super, super simple, but there are going to be times when you 're working with data and it's not going to be so evident, right? It's going to be a little bit more fuzzy. But that's why these foundations are crucial because whatever data set that you 're working on, whatever problem you're trying to solve or trying to understand these foundational concepts will help. That's the takeaway. Before we end this video, I did want to show you how to get into Google Sheets. And so the way that I did this is I went over here to docs.google.com/spreadsheets, but you can also just go to Google Drive and then click on spreadsheet. So what I'm going to do is click on this blank spreadsheet here. And so here we have a spreadsheet. And this is going to be part of the challenge to the very end. So what we're going to do is present you with this challenge involving a spreadsheet. So let me just show you how these things work.

4:34Let's say that we had, let me add a number here, and then I'm just going to drag this down. And then it's going to give us those numbers. But let me copy this or maybe I can drag it this way too. And so we have some numbers. It's not important what the values are. But let's say that I wanted to reduce the dimensions and I could cut this right here like this. Right? And then so I've removed that. But you can also delete the column. So delete that column. And so now I've removed the dimension. So we went from four dimensions to three dimensions. So just wanted to give you a little bit of context so that you could feel confident going into the challenge. And I will see you in that video next.

0:00Welcome back and congratulations on making it to the challenge. And the way that this works is the challenges are going to be an interactive step, right? To make sure that you have all of the concepts down and for example, we might do something like use Google Colab and you work with data or in this case we're going to work with a spreadsheet. And we already talked about how to do that, ignore that. You would go over here and go to new and create a spreadsheet or if you have a CSV file, which I'm going to provide to you, you can use import to import that CSV file, which is exactly what I did and then we have this.

0:36So that's your first step is to take that CSV file, which is a comma separated values. And I'll show that to you. Once you have that, then you import it and then you 'll get this. And then what you have to do is answer questions about this. So we're going to talk about data types. We're going to talk about dimensions. We're going to talk about everything that we talked about or at least test your knowledge of all of this. But the good news is that this is not graded. So this is just an opportunity for you to get the juices flowing into apply these concepts. So I will see you in the next video where I show you the basically how I imported this and talk a little bit about the answers for each one of the questions. See you there.

0:00Welcome back and congrats on completing the challenge. So let me walk you through the steps of how I completed this. The first thing is I had to download the CSV file that's going to be located right above this video in the challenge instructions. And once you download that then you go to File and then you go to Import and then you go to Upload and then you find that file and then upload it . I've already uploaded to my drive so I'm just going to click this and then I'm going to click on Insert and I'm going to say Important Data Adomatically. I have changed anything and then once that's done, it might take a second but then you get this. So this is that Untitled spreadsheet and now we have all of these values and how many values do we have? Quite a few. Alright, maybe a thousand.

0:49So we have Horsepower, Miles Per Gallon, Weight and Cylinders. Now there's no right or wrong answer here. This is except for loading the data. You got to be able to see this. So outside of that this is more about just giving you an opportunity to experience the different data types and to think about it. In the first question is what kind of data type are the features here? This is Horse power, Miles Per Gallon, Weight and Cylinders. So we talked about the two different kinds of data types which is qualitative and quantitative data. So you have to answer what type of these? What type of data structure are these? And the easy answer is they're not numbers so they're going to be qualitative. These are going to be colors, names, labels. So these are labels for the data below. Then the second question is Horsepower. What are the values here? What kind of data type is that? So that one is clearly quantitative, quantity because you can count it, measure it, right? So that's what we're doing here. Horsepower and Miles, this Horsepower are going to be quantitative. And to be quite honest they're all

2:07quantitative. These are all numbers, right? So all of the values for each one of the features are going to be quantitative. And since this is raw data it's not directly useful but for this question we've been given a task to provide some sort of data for an article for a car magazine. And then we want us to compare the fuel efficiency of different vehicles. So we have to pick only two features. So which two features would you pick? However the question is really focusing on the most important feature. So just go ahead and enter Miles per gallon. And so once we have the Miles per gallon we can use any of these others to calculate fuel efficiency. I think the Miles per gallon is probably the most important and I would say either weight or cylinder but this is all subjective. The most important one is definitely Miles per gallon. And for the final one we've decided that we're going to reduce the dimensionality and we're going to include only Miles per gallon and cylinders. So we need to reduce the dimensions. The way that I did that is I go to data or in the safe which where is it? So I right click on this

3:27feature then I find delete column. There it is. Delete that column and we're gonna do the same thing. Yes I did that on purpose. And then here we're going to do the same thing and delete column. Oh no I need to select just one and delete column. Okay so now we have reduced our dimensions and now we can have or study the relationship between Miles per gallon and cylinder. The other data point that we're not talking about is like why kind of car but let's just imagine that those are handled after the fact once we know what are the most fuel efficient we'll probably pick the top five then we'll look at this index and find out what the cars are. So that's just not provided in this case but that's okay because we don't always have the complete data. But congratulations on completing this challenge and I will see you in the next skill. Until then I hope this has been informative and I'd like to thank you for viewing.