Data Science
Where can I find a repo of "dirty" datasets to practice feature engineering?

submitted by /u/Mah_Legzz
[link] [comments]

Data Science
Any ideas for state space models master thesis?

I have to write a master's thesis (60-70 pages) for my AI Msc. I have soon learned about sparse state space models and find them interesting. I am also interested in stock price prediction. Now I am reading articles but find it hard to propose some novel idea. How do I do it? Do any unsolved problems in that field exist?

submitted by /u/Pineapple_throw_105
[link] [comments]

Data Science
Solo preguntas que algunas personas se hacen

Alguna vez alguien se a puesto a pensar que un dejavu puede ser una conexión de nuestros serebros entre dos realidades como si dos universos nos dieran a vivir el mismo momento con una pequeña diferencia en el tiempo

submitted by /u/Zcris21
[link] [comments]

Data Science
Is it worth it?

I finally scored an offer as a new grad in this market, as a pricing data analyst (well, its moreso data-engineering)! I've been given a 2 week deadline to accept it. The reason I haven't yet to accept is, I find this place shady:

* This business is tiny (well, it's really a local consultant business contracting for a bigger business). Everyone's remote and the "HQ" address is a random residential suburban house. The linkedin page has like, just 6 people.
* As a result, there's none of the typical HR stuff I'd usually expect to see and sign when starting the job, because there's no HR at all. I'm expecting the true legal stuff begins after I join but coming from the bigger company that this business is contracting with?
* It's hybrid, but the first few weeks is said to be fully remote before they can fully set me up...and I'm expected to use my personal PC but via remote desktop (is this not a red flag?).
* It's 29$/hr or 60k/yr with medical benefits. But how are they paying that? Where are those money and benefits coming from? the bigger company? They told me the role is "full-time" but this feels like a contract role, no?
* On the other hand, the business has their own website and linkedin page. I had video interviews with the people, so I've seen their faces and voices. They are in fact real people. I did some more sleuthing based on their linkedin pages, and they do in fact seem legit. i.e. masters degrees, decade-long experience at many companies, connections, endorsements, etc. In other words, this is 100% not a job scam obviously.

It's hard to tell if this place is sketchy ...or just tiny and obscure. Does this pass the smell test? Would I be foolish to decline it as a new grad with just a data science BS and 0YOE in this market? Should I still take it, but job-hunt on the side?

submitted by /u/newauthry
[link] [comments]

insideBIGDATA
Infosys Generative AI Radar Report

https://insidebigdata.com/wp-content/uploads/2023/06/GenerativeAI_shutterstock_2284999159_special.jpg Our friends over at Infosys Knowledge Institute released its Generative AI Radar report providing insight into over 1,000 U.S. and Canadian businesses on the current state of generative (gen) AI implementation and early indications of its ability to deliver value.

Data Science
What are your duties as a Data Scientist?

Please elaborate on this.

Including your role at the company, your day to day tasks, tools and languages you’re using.

Thank you in advance!

submitted by /u/Judessaa
[link] [comments]

Data Science
Is there a way to find how many python toolkit downloads in a certain period of time?

I was wondering if there was a way I could find how many downloads there were of NLTK vs. spaCy in 2023. If anyone can help me I would greatly appreciate it! This will be very helpful, much obliged. Thank you and have a good weekend, cheers.

submitted by /u/Fun_Crab8862
[link] [comments]

Data Science
PSA: Don’t become DS. Be a DA instead.

I’ve been on this board for a few years and noticed a trend. Many people saying they got a MS in DS and complain they only do excel or simple models. Recently, I see a lot of people saying they can’t get DS jobs. Here is the thing, most businesses need a lot more DA then DS. There are so many more basic data needs then complex ones. Most companies I’ve worked for have a ratio of about 5:1 DA to DS. Unless you’re a really strong and savvy DS candidate (smarter then me) you’re probably better off doing DA or SWE. I am a DS director and I spend 80% of my time doing DE and DA because that’s what the business needs.

submitted by /u/NewEcho2940
[link] [comments]

Data Science
What is the worst case of PHDitis that you have seen?

Title

submitted by /u/blacksnowboader
[link] [comments]

Data Science
[Meta] New Automod Rule - Minimum Comment Karma before Submissions

After feedback from many members and discussions within the mod team, we have decided to implement a new Automod rule:

Rule: Effective immediately, users must have at least 10 comment karma within r/datascience before they can make a top-level submission.

The desired outcomes are:

1. Reduce pure self-promotion botspam
2. Reduce the number of top-level submissions that belong to the Weekly Sticky thread.

Please let us know if it appears to be working incorrectly or causing unwanted side effects.

submitted by /u/Omega037
[link] [comments]

Data Science
Recently Accepted into Masters Program and Seeking Views on Job Market

Hey everyone, I've recently been accepted into Georgia Tech's OMSA program following my undergrad in health sciences as I was hoping to make a pivot into DS as I believe it would fall directly into my realm of intrigue in terms of statistical analysis as well as with a passion towards working in tech. However, I've seen countless posts about how brutal the job market is for DS and that finding a job is incredibly difficult and it is nowhere near as lucrative as it once was which has me worried and potentially regretting my decision. Prior to pursuing DS I was going to be pushing forward in a career towards dentistry but the cost of tuition and long years of schooling kept me away from that. I'm just worried as if I'm making a poor choice in terms of my future regarding to career path and if this masters will even help me in the market.

submitted by /u/Marcus_m21
[link] [comments]

Data Science
Learning resources for a new DS manager?

Tl;dr - Soon to be transferring over to a DS manager role from an analytics manager, and I do NOT want to be that leader. What are some recommended MOOCs, videos, books that can boost my technical knowledge over the next 2-3 months.

I have been on the analytics side for ~10 years, and have a strong foundation of SQL, python, data viz, and analysis, and a solid knowledge of math/stats (can still be improved). I’m lacking in the ML and deployment space, and have a couple months to study up here. Any strong recommendations of courses, videos, or problem sets to work through? (Books are also great, but I am painfully slow and may be more efficient with another medium). Thanks in advance.

submitted by /u/stryder517
[link] [comments]

Data Science
What would you classify my job as? DS, DA, DE, Glorified Excel Monkey

Officially I am a Data Scientist. I try to understand my value or worth outside of the government.

What I don't do: AI, ML, modeling.

What I do: Develop new data pipelines, Data exploration, Produce data and dashboards from policy and new concepts, Python, R, SQL, Databricks.

I feel a DS should be doing ML at minimum but our business needs are fast and dirty and the data is dirty. Dirty data = Dirty results is how I view ML stuff.

Edit: Punctuation because I forgot about Reddits mobile formats lol

submitted by /u/Yourteararedelicious
[link] [comments]

Data Science
D.S. / Analytics Directors, What Tools Do You Use to Organize Your Work & Knowledge?

Curious what people are using to keep track of projects and general data/process documentation

submitted by /u/whispertoke
[link] [comments]

Data Science
How’s the DA job market looking for people with experience?

I’ve started applying around for data analyst roles this week and was wondering how people with 1-3 years experience are doing with their job searches

Asking since most posts on here are either like “no experience how do I break in” posts or like PhD data scientists with not much in between

submitted by /u/evavibes
[link] [comments]

Data Science
Taking over new role as DS manager

Not sure if the topic is allowed, but I would like to take opinions from senior data scientists and Analytics managers.

I work in finance as data scientist and work involves preparing data in required format, Doing analysis and building models for products that we have in market , like propensity models for credit cards , credit risk models etc. I have worked as an individual contributor till now and have 5 years of experience. I have never managed anyone but have mentored and led few projects individually. I have a new offer for an analytics manager with a well known bank and I'll have to manage 6-7 data analysts/scientists and be responsible for the team’s performance.

The pay jump is decent (40 percent higher than I currently make) and location is much closer to my home. I don't have any problems with my current job and people I work with are also great.

I was thinking if anyone else made that jump. Is the transition too steep from not managing anyone to 6-7 people?

submitted by /u/shar72944
[link] [comments]

Data Science
Need a friend, interested people please dm

Hi clan, I am a data analyst and currently pursuing a distance masters program in data science and machine learning. But unfortunately, I have never been a classroom learner, and always fail miserably while following classroom teaching. Although I found out, what keeps me enticed is project based learning where , by building new stuff, I learn new things.

But being a distance learner, it gets pretty hard to stay motivated and work on projects solo. Recently I came up with concept of 42 school, France, where a group of like-minded people would work on projects together and learn along the way in a hands-on approach. Long term, I think I would like to build a peer based learning community in data science, where students would learn from each other instead of sticking to any fixed curriculum being delivered by any teacher per se.

But , ideas can be wild, so before building this community , I want to test this approach on myself to see if I can learn in a similar way first. For that, I would need a partner (or two, or three, the more the merrier I guess) to start on this journey.

What the other person would get from this are -
1. An accountability partner.
2. Peer based complimentary learning. ( where we can explain and teach topics to each other)
3. A group to participate in hackathons and do projects together.
4. And last but not the least, some friends, who are on the same path.

If you have any questions for me, please feel free to reply to this thread, I will try my best to answer them. If you are interested in this experiment and want to join, either you can dm me, or can leave a reply to this thread.

P.S: Please don`t think me as a fake/bot profile due to my low karma, I am mostly a silent browser of reddit and haven`t been active in periods in between.

submitted by /u/69casual_dreamer96
[link] [comments]

Data Science
Guesstimates

Is there any book , podcast y'all can recommend to study guesstimatation problem

submitted by /u/charlesowo445
[link] [comments]

Data Science
Python package for statistical data animations

https://external-preview.redd.it/MoP6enMQ2Q6o4o23d5xCmvlBtpeCXWiqxc63UVCX5Rk.jpg?width=216&crop=smart&auto=webp&s=cfd7f76ac4c13cdc287edd9856ef0430dbc862a5 Hi everyone, I wrote a python package for statistical data animations, currently only bar chart race and lineplot are available but I am planning to add other plots as well like choropleths, temporal graphs, etc.

Also please let me know if you find any issue.

Pynimate is available on pypi.

github, documentation

Quick usage
import pandas as pd from matplotlib import pyplot as plt import pynimate as nim df = pd.DataFrame( { "time": ["1960-01-01", "1961-01-01", "1962-01-01"], "Afghanistan": [1, 2, 3], "Angola": [2, 3, 4], "Albania": [1, 2, 5], "USA": [5, 3, 4], "Argentina": [1, 4, 5], } ).set_index("time") cnv = nim.Canvas() bar = nim.Barhplot.from_df(df, "%Y-%m-%d", "2d") bar.set_time(callback=lambda i, datafier: datafier.data.index[i].strftime("%b, %Y")) cnv.add_plot(bar) cnv.animate() plt.show()
​

https://i.redd.it/27xu9yip74xb1.gif

A little more complex example

​

https://i.redd.it/kycvoy4u74xb1.gif

(note: I am aware that animating line plots generally doesn't make any sense)

submitted by /u/julkar9
[link] [comments]

Data Science
The job market is so frustrating

Graduated 5 months ago with a MS in CS. Before I came to the US to pursue my masters, heard from a boat load of people that getting jobs after graduation was easy and that hardly anyone graduated without a couple offers in hand. That sentiment was echoed by other recent grads I met when I got here.

I always wanted to get into DS, so when everyone started looking for internships, I started looking for DS/DA/DE internships specifically. Gave a bunch of interviews, landed an offer in April of 2022. Just an unfortunate decision. The company had a new data science practice with no clear definition of what a Data Scientist does. Being a consulting firm, we basically jump from one case to another and use whatever tech is needed on a case to case basis. Spent all summer just doing web scraping and OCR extractions. Also, my manager is super condescending and outright rude. He’s told me multiple times that he “can’t believe I have two degrees in Comp. Sci” and at team gatherings and social events, wouldn’t even look me in the eye or acknowledge my existence lol. On the last day of my summer internship, he was in my office literally laughing at my code which btw was based off a snippet he sent me.

Anyway, once this ordeal was done, the world went into a recession and I had to accept a return internship offer. Return internship because I hadn’t proven myself enough to land a full time role yet. Went through another 3 months of abuse and got a full time offer, been working FT for about 4-5 months now.

At this point I can’t take it anymore. Every day at work I’m putting out fires with the fear that if I fuck up, I’ll either be publicly ridiculed or fired. Consulting being consulting, work life balance is non existent and I had to move to a city where I have no friends and no social life to at least escape the stress.

To all seniors and hiring managers etc, do you think the job market is going to get better? What’s the trend at your company?

submitted by /u/Objective-Test5021
[link] [comments]