Quick Analysis with Python

Picture this scenario: Your boss sends you an email with a task they need you to complete. What is the task? You open the email and notice a large attachment along with a short note. The attachment is a large data extract (.csv) with hundreds of thousands of rows

Picture this scenario: Your boss sends you an email with a task they need you to complete. What is the task? You open the email and notice a large attachment along with a short note. The attachment is a large data extract (.csv) with hundreds of thousands of rows, and the note reads, 

“Can you give me the correlation coefficient for variables A & B in this dataset?”

TACKLING THE PROBLEM

You’re a problem solver, so you quickly download and open the attachment with Microsoft Excel, attempting to tackle the problem head-on. You find columns A & B to perform the correlation; then you start typing in a new cell, =CORREL(arrayA,arrayB), and press ENTER. The Excel formula returns a correlation coefficient, but it can’t be that easy, right? That’s because it probably isn’t. There are other variables in this dataset that need to be considered in order to produce the best results, so in order to tackle this problem, you must break down those other variables as they relate to variables A & B. In other words, we need to partition the correlation of variables A & B on variables C, D, E, F, etc. Now, that all sounds very abstract, so let’s dive into something more concrete.

USING THE RIGHT TOOLS

For example, let’s say you’re looking to correlate all of your utility’s load profile data with temperature/weather data. You’re expecting a strong positive correlation because as it gets hotter, people tend to use more electricity. The issue, however, is that your utility supplies residential, commercial, and industrial customers. The usage pattern for a common residential household will vary significantly from that of an industrial plant, so in order to get the most accurate correlation, you need to partition your analysis on rates, customer class indicators, seasons, and potentially even location (e.g. zip code). Unless you really want to go through hundreds of thousands of rows of data to manually perform this kind of partitioning, Excel is probably not the best tool to use for your analysis. Why work harder than you need to when there are other tools available to help you work smarter?

Look no further than Python, as there’s an abundance of capabilities that you can now explore. Consider using libraries such and pandas, numpy, and matplotlib to read your data, perform correlation calculations, and plot graphics. You can group your data by whatever variables you’d like (e.g. rates and zip codes) and write loops to quickly and efficiently to perform all of your analysis and print the results. Once you’ve written your program, simply save the output, send it back to your boss, and your work is finished. You now have some robust and reusable code in your arsenal, and on top of that, you learned a new skill.

LEARNING SOMETHING NEW

While self-quarantining and social distancing are in full effect thanks to COVID-19, you might have some extra time on your hands, so consider taking full advantage of that time by tackling a problem that calls for learning a new toolset. Maybe there’s a problem that you haven’t quite gotten to yet, or maybe you’re consistently tackling the same problem by way of brute force. Either way, now might be the best time to try a new tool, one that is more appropriate to the problem at hand and can be kept in your toolbox for others that arise in the future.

—————————————————–

Follow QBA on social media: LinkedIn | Twitter | Facebook

113 thoughts on “Quick Analysis with Python”

  8. I simply couldn’t depart your web site prior to suggesting that I actually enjoyed the standard info a person supply on your visitors? Is gonna be back steadily to check up on new posts.

  9. I just could not depart your site before suggesting that I actually loved the standard information a person supply to your guests? Is going to be again often to check out new posts.

  10. I simply couldn’t depart your site prior to suggesting that I really enjoyed the usual info an individual provide to your guests? Is gonna be back ceaselessly to investigate cross-check new posts.

  11. Good day! This is kind of off topic but I need some advice from an established blog. Is it very difficult to set up your own blog? I’m not very techincal but I can figure things out pretty fast. I’m thinking about creating my own but I’m not sure where to begin. Do you have any points or suggestions? Thank you

  12. When I initially commented I clicked the “Notify me when new comments are added” checkbox and now each time a comment is added I get three emails with the same comment. Is there any way you can remove people from that service? Bless you!

  13. My brother suggested I might like this website. He was entirely right. This post actually made my day. You can not imagine just how much time I had spent for this info! Thanks!

  19. Woah! I’m really loving the template/theme of this
    blog. It’s simple, yet effective. A lot of times it’s challenging to get that “perfect balance” between user friendliness and
    appearance. I must say that you’ve done a excellent job with this.

    Additionally, the blog loads extremely fast for
    me on Opera. Exceptional Blog!

  21. It’s appropriate time to make some plans for the future and it’s time to be
    happy. I have read this post and if I could I wish to suggest
    you few interesting things or advice. Maybe you could
    write next articles referring to this article.

    I wish to read even more things about it!

  24. It’s perfect time to make some plans for the future and it’s time to be
    happy. I have read this post and if I could I wish to suggest you some interesting things or
    advice. Perhaps you can write next articles referring to this
    article. I desire to read more things about it!

  26. I will immediately grab your rss as I can not to find your e-mail subscription hyperlink or e-newsletter service.
    Do you’ve any? Please permit me know so that
    I may subscribe. Thanks.

  28. I think this is one of the most important info for me.
    And i am glad reading your article. But wanna remark on few general
    things, The site style is great, the articles is really excellent : D.

    Good job, cheers

  29. I absolutely love your site.. Great colors & theme.
    Did you build this amazing site yourself? Please reply back as I’m trying to create my own personal website and would like to learn where
    you got this from or just what the theme is
    named. Kudos!

  30. Purely to follow up on the up-date of this theme on your web page and would want to let you know how much I treasured the time you took to produce this beneficial post. In the post, you spoke of how to seriously handle this problem with all ease. It would be my personal pleasure to build up some more ideas from your web page and come as much as offer other individuals what I have learned from you. I appreciate your usual terrific effort.

  32. I am sure you will like this 1website

    k23lartds11
    The game is more fun when you have unlimited diamonds.If you like mobile games like this you need to check out this link

  33. Pingback: sildenafil mexico price

  35. It is perfect time to make some plans for the future and it’s time to be happy.

    I have read this post and if I could I wish to suggest you
    some interesting things or tips. Perhaps you can write next articles referring to this article.
    I desire to read even more things about it!

  39. Greetings! Very useful advice in this particular article!

    It’s the little changes that make the largest changes. Thanks a
    lot for sharing!

  43. It is perfect time to make a few plans for the longer term and it
    is time to be happy. I’ve read this post and if I may I want to counsel you some
    fascinating things or advice. Perhaps you could write subsequent
    articles regarding this article. I want to learn more things about it!

  48. I don’t even know the way I finished up here, however I assumed this put up was great.
    I do not understand who you might be but certainly you’re going to a famous
    blogger if you are not already. Cheers!

  52. You can certainly see your enthusiasm in the article you write.

    The arena hopes for even more passionate writers such as you who are not
    afraid to say how they believe. At all times go after your heart.

  54. Terrific work! This is the type of information that are meant to be shared across the net.
    Disgrace on the search engines for now not positioning this submit upper!
    Come on over and discuss with my site . Thank you =)

  60. I’ve been surfing online greater than three hours nowadays,
    but I never discovered any fascinating article like yours.
    It’s lovely worth enough for me. Personally, if all webmasters and bloggers made good content as you probably did, the internet shall be much more helpful than ever before.

  62. Very good site you have here but I was wanting to know if you
    knew of any user discussion forums that cover the same topics discussed in this article?
    I’d really love to be a part of community where I can get feed-back from other knowledgeable individuals that share the same interest.
    If you have any recommendations, please let me know.
    Thank you!

  63. 1Learn more about how to get free diamonds on cooking fever

    u3iauds11
    Cooking Fever is a lot more fun when you have unlimited gems.If you love mobile games like this you should check out the guide above

  68. Wow that was unusual. I just wrote an very long comment but after I clicked submit my comment
    didn’t appear. Grrrr… well I’m not writing all that over again. Anyways,
    just wanted to say great blog!

  70. I’ve been surfing on-line more than 3 hours lately, but I never found any attention-grabbing article like yours.
    It’s beautiful price enough for me. Personally,
    if all web owners and bloggers made excellent content
    as you did, the net might be a lot more helpful than ever before.

  73. Hi I am so grateful I found your webpage, I really
    found you by mistake, while I was researching on Bing for something else,
    Regardless I am here now and would just like to say cheers for a remarkable post and a all round
    enjoyable blog (I also love the theme/design), I don’t have time to browse it all at the
    minute but I have saved it and also included your RSS feeds, so when I have time I will be back to read a lot more, Please do keep up the excellent b.

  74. Hello to every body, it’s my first pay a visit of this website; this website carries awesome and in fact good information for visitors.

  81. Hi, I do believe this is a great site. I stumbledupon it
    😉 I am going to revisit once again since I book-marked it.
    Money and freedom is the best way to change, may you be rich and continue to guide others.

  84. I like what you guys tend to be up too. This type of clever work and reporting!

    Keep up the amazing works guys I’ve included you guys to our blogroll.

  87. Pingback: cephalexin cost
  88. Pingback: bactrim kidnet

  92. In accordance with my observation, after a the foreclosure home is marketed at a bidding, it is common for any borrower to be able to still have some sort ofthat remaining unpaid debt on the loan. There are many creditors who attempt to have all charges and liens repaid by the up coming buyer. Even so, depending on specified programs, rules, and state laws there may be many loans that are not easily resolved through the exchange of loans. Therefore, the responsibility still falls on the lender that has received his or her property in foreclosure. Thanks for sharing your opinions on this blog site.

  101. Pingback: metronidazole odor
  102. Pingback: zithromax tablets
  103. Pingback: 500mg azithromycin
  106. Pingback: augmentin espanol
  107. Pingback: clindamycin taste

  108. I will immediately grab your rss as I can’t in finding your e-mail subscription hyperlink or
    e-newsletter service. Do you have any? Kindly allow me recognise so that I may just subscribe.
    Thanks.

  110. It is appropriate time to make some plans for the future and it’s time to be
    happy. I have read this post and if I could I want to suggest you few interesting things or advice.
    Maybe you can write next articles referring to this article.
    I desire to read even more things about it!

Comments are closed.