Comments on: Results: Identical Ad Sets, a Split Test, and Chaos

By: Benjamin Völkel

Benjamin Völkel — Mon, 07 Oct 2024 08:47:57 +0000

I agree on this. Currently I have the same impression that META cannot provide fair AB-test groups. Groups seem to be significant different regarding auto-played-activated or -deactivated. This screws up heavily the conversion results. People with auto-play-deactivated click on ad because they just want to get the video played. That is not a real install-conversion…

Would be great if @jonloomer:disqus could look into this within his META test result.

By: Branko Boro

Branko Boro — Mon, 07 Oct 2024 03:50:15 +0000

This comes as no surprise to me. As my clients generally don't have the budget to run significant tests I just make sure to give the algo enough options to play with.

By: dennisDenjo

dennisDenjo — Fri, 04 Oct 2024 09:17:38 +0000

Hi Jon,
Is it possible for you to see the share of "auto-played" and "Clicked-to-play" Videos per ad? (settings of facebook users) I had a case once where this split differed a lot and was most likely the driver for different results

By: Adrijus Guscia

Adrijus Guscia — Tue, 01 Oct 2024 20:10:35 +0000

In reply to Jon Loomer. I bet you would in some cases, but in some cases you would get better results and differences. Back some years you could even use 3rd party tool to see which keywords worked best. And I miss those days...

By: Jon Loomer

Jon Loomer — Tue, 01 Oct 2024 13:29:29 +0000

In reply to Adrijus Guscia. If you were to run this test while having complete control over targeting, I would expect you'd still get similar randomness. People aren't robots. When you're dealing with smaller sample sizes, you can't guarantee exactly same results from exactly same ad sets.

By: Jon Loomer

Jon Loomer — Tue, 01 Oct 2024 13:28:25 +0000

In reply to Jelle. But that again goes into randomness that we need to consider.

By: Jon Loomer

Jon Loomer — Tue, 01 Oct 2024 13:27:21 +0000

In reply to mi mi. It's a split test, so I'm using ABO. The amount spent was almost exactly the same for all three.

By: Adrijus Guscia

Adrijus Guscia — Tue, 01 Oct 2024 13:05:14 +0000

Honestly, this just makes me trust Meta ads less… they were good with Interest targeting, but now the algo might just spend the money uselessly. Don't trust AI (or more like greed in general) as you have little control anymore and it was awesome for that reason.

By: Jelle

Jelle — Tue, 01 Oct 2024 11:01:10 +0000

Thanks for the interesting read. Did you also check on country with the sets A, B and C? Maybe B started to target more people in one country and the ad was more succesful for the people living there?

By: mi mi

mi mi — Tue, 01 Oct 2024 10:51:34 +0000

hi! are you using CBO or ABO?

By: Stefan

Stefan — Tue, 01 Oct 2024 02:13:20 +0000

Why I love you, this is
Wonderful approach to show something important, it is.
Listen and learn, you must.
/Yoda