Видео

How would you do if you want to perform an A/B/C test?

ChatGPT can answer all the question related to code. So I don't think interviewer should ask those questions anymore. They would focus on how we solve the problems by ideas

Walk. Physical

Is the expected output usually showed in the interviews?

Thank you for creating this video. It provides such a clear and concise overview of A/B testing 👏

Hi I'm on the website but I can't find the corresponding feedback video

should the variance not be multiplied by 2?

The video is incorrectly labeled market sizing - the more accurate description should be Measuring Success

I excelled the SQL but couldn't ace the Python part

I think the candidate was nervous and spike before thinking many times in the product case. Maybe if she had a couple minutes to compose her thoughts, she coulda done better.

👍thank you so much

Both the interviewer and candidate were pretty bad here. Candidate was low energy, and the interviewer didn't really dive deeper... just asked surface level questions and moved on

@datainterview Wouldn't the click through rate or number or products purchased per user per day be a more effective success metric? If it is revenue per user per day, what if the control group purchases only one high priced product whereas treatment group purchases multiple low prices products, the recommendation system works but the revenue would be higher for the control group

1. Looking at the past history of the same airline, the preliminary cause of the flight delay might be operational issues. 2. Geography 3. Weather 4. No of runways - if they are too less then if one flight gets delayed then the chances of others getting delayed increases

what software/ tool is used for the A/B testing example in the video?

idc about "acing" anything

6:43 should the numerator be cov(X,Y)? Seems there is a 1/(N-1) term missing.

Good video, but I think you forgot to ask the most important question: how much money the business can save with your model and whether it is worth implementing it in the first place. In your case, you want to create a model that would predict delays in real time and for passengers that are already in the airport and their departure should be in the next couple of hours. But in this case, either your prediction won't be important at all (if it is 10 min or 1 hour delay), since passengers can't do anything with this information, or your delay prediction is so large (5+ hours) that users will ask for extra compensation (since you didn't inform them earlier) and complain why didn't you inform them a day before that the plane is delayed by 5 hours. In your case, you assumed it is for people who are already at the airport, but I would argue that the actual end users are people who's flight is in a day or so. Since if the company would inform users a day before that their flight is 5 hours delayed, users can make adjustments and might not complain so much and request for a refund, compare to when you predict that the flight will be 5 hours delayed when the user is already on the airport. It's more important to predict whether a flight will be delayed 12+ hours before departure, so that the passenger would decide for themselves when to leave home and whether to make other plan changes, rather than a couple of hours before the departure, when most probably everyone is at the airport, and the delay can be easily calculated using a simple math. And when people are already on the airport, it's already too late to tell them that their flight is 24+ hours delayed. In this case you don't need to have a real time prediction. Next point, I don't agree with 95% cutoff. Yes, there might be some bugs, but I would not cutoff like that, since those flight delays might be actually the most important ones (which have 24+ hours delay)

Question on metric tested in the hypothesis test - avg. # of clicks per user. Why wouldn't it be avg revenue per user per day? I believe the user could click the "buy now" button and still exit, or they may buy but spend less money with the new button,etc....

Excellent video. Should the candidate list the guardrail metrics way before step 7 (i.e., step 1) so those are known & agreed upon before the test starts ?

This is a neat solution but can be made 5x more efficient if you map the sum of the two rolls to an outcome on the seven sided die: For e.g Sum Number on 7 sided die (2,5) -> 1 (3,4)->. 2 (6)->. 3 and so on (you will need to remove one of the 6 cases which give 7 though). This way, we will be using 35 out of 36 possible outcomes instead of just 7

Bro I have an interview in the morning and your video is giving me the confidence I need to go through with it. 😅

Thanks for sharing! This video on A/B testing is hands down the most informative and practical one I've seen on RUclips.

the best a b testing video

What if we have same user making multiple purchases across the day? Does it make sense to have metric as revenue-per-session / revenue-per-search if we have multiple purchases by same user in a day, and metric is revenue per user per day, wont we have to change the randomization unit?

Hey Dan, can you talk about the decision to pick revenue-per-user-per-day? Doesn't using this metric require the delta method or bootstrapping if randomizing by user? Thanks!

Thank you very much for explaining this problem.

Why for the second question we use subquery instead of using joins and put filter in the end?

I am sorry, I have found a lot of good content in this channel but this video has to be the worst one.

Am I mistaken, or is the description of the Type II error at 8:05 incorrect? He says Beta is the "the probability of rejecting a null hypothesis when the alternative hypothesis is true." But isn't Beta/Type II error the probability of *not* rejecting a null hypothesis when the alternative is true? Genuinely trying to clarify to make sure I have proper understanding.

This is the only video one would needs to demystify the power analysis. Thank you.

Why does every one just start writing the equation? Where is the deeper intuition?

What about a high NDCG score ?

What is lift?

Oof so many dead data science channels now.

Q1: (n*(n+1)/2)/100

I feel like the dude got lost in the sauce with seller based, listing based type shit.

The interviewee does not understand what he said. Unfortunately, he now becomes interviewer. This explains why so many people who knows these stupid stuff failed the interviews.

SELECT AVG(items) AS average_order_size FROM Customer AS c JOIN Transaction AS t ON c.transaction_id = t.transaction_id; I optimized the SQL query. Can't we use this?

what if we calculate the confidence Interval for Population Proportion using the standard error of the estimate using sqrt(0.7*0.3/100) and check whether 0.5 lies in the resulting Confidence Interval?

Hey Dan, really great video. One question. What will be the possible metric if we want to test a recommendation system for a fee i.e. for items shown on homepage rather than when search happens.

For the t-test, since this is a multivariate test, wouldn't we apply Bonferroni correction?

Incredible! Professional yappers, very useful for practicing corporate English✍️😁

It is a great video. Even for an experienced DS, it helps to organize thoughts before interviews.

This was really good

1:08 8 hours a day in Facebook????? What is the X at the bottom?

great lecture and thanks for sharing. But why is the two-sample pooled variance for proportion is the sum of the two samples's variance? should it be the 2*variance 1, because of the similar reason to it's mean counterpart?

First I will list the variables, then use the data to test which of those are statistically relevant. Then from those that are relevant which is the most confident source of information, can we determine seasonality. Which of those variables are under our control. Then I will propose a couple scenarios of Failure Mode and Effects to observe in more depth the problem. Based on that run experiments with existent data to understand better in practice how it looks like and propose high level scenarios.

with cte as ( select user_id, payment_method, count(*) as cnt, min(transaction_dt) as first_trns_dt from amazon_transactions group by user_id, payment_method), cte_2 as ( select user_id, payment_method, cnt, row_number() over(partition by user_id order by cnt desc, first_trns_dt asc) as rank from cte where cnt > 2 ) select email, payment_method, cnt as payment_counts from cte_2 c2 join amazon_users au on c2.user_id = au.user_id where rank = 1 order by payment_counts desc, email asc ;

Hi Dan! Thanks for this great video. In the end, when you compare practical significance, do you compare the experiment Lift value with the confidence interval? Is this possible even that the formula of confidence interval uses absolute difference?