Treasure Hunter Proc Rate Testing -TH 8 Versus 14

Hi all. Many of you on the the thief forms are aware that I started collecting data samples on treasure hunter proc rates with two different sets- one with worn treasure hunter 8, and a second with worn treasure hunter 14. The goal of the test was to determine whether or not wearing more than treasure hunter 8 in gear has any effect at all on the rate at which treasure hunter levels up via melee procs. I started collecting this data about a year ago but due to the amount of time it takes to collect a large sample size I stopped gathering numbers midway. This data has been referenced several times since, and there's has been a perpetual disagreement as to whether or not wearing more than TH + 8 does anything at all. Well, the discussion was rekindled again on the sticky so I decided to go out and finish gathering more data. I spent the better half of a day today collecting a large enough sample size, but I think there's enough data here to make some good assessments.

Things of note -- Originally I was using Perfect Taming Sari, Sandung, Gorney Ring, Chaac Belt, Plunderer's armlets +3, and skulker's poulaines +1 to achieve treasure hunter 14. Since then I've obtained a second perfect taming sari and tossed my old sandung. I've also incorporated a perfect lucky egg into the TH 14 set, so in actuality the new Treasure hunter + 14 set is using Treasure hunter + 15 instead. I denoted on my spreadsheet where this point occurred.

In a similar vein I also updated my Treasure hunter 8 set. Instead of dual wielding Twashter/perfect taming and using plunderer's armlets +3 and gorney ring to achieve treasure hunter 8, I'm now using Twashter/Gleti's knife and using the armlets combined with perfect lucky egg instead. You'll be able to see this in the numbers because the increased DPS results in approximately 10-15% fewer swings per mob.

I'll be posting the numbers and doing a breakdown analysis and summary below.... however if you're only interested in the conclusion and want the TL;DR version here it is.

According to the numbers I've gathered wearing more than treasure hunter + 8 has no noticeable difference on proc rate, and the difference between Treasure hunter + 14 versus Treasure hunter + 8 is completely irrelevant, and probably nonexistant. Worn Treasure Hunter values appear to cap at Treasure hunter +8, and wearing any more than that has NO benefit whatsoever to treasure hunter proc rate.

Now with that said, here's the summary below.

-------------------------------------------------

The first thing I'll do is post the data. I've saved it in a google spreadsheet which can be found here.

So to start off with I wanted to aggregate some of the data. I'll post that below for both the Treasure hunter 8 and the treasure hunter 14 sets


Treasure Hunter 8
---------------------------
Total Mobs: 40
Total Treasure Hunter 9: 4
Total Treasure hunter 10: 16
Total Treasure hunter 11: 15
Total Treasure hunter 12: 5
Total Treasure hunter 13: 0
Total Treasure hunter 14: 0

Total Treasure Hunter Procs: 101

Total attacks: 23,411
Avarage attacks per mob: 585.275
Average Treasure Hunter procs per mob: 2.525

Average total attacks to get a Treasure Hunter proc: 231.792

Treasure Hunter 14
---------------------------
Total Mobs: 40
Total Treasure Hunter 9: 2
Total Treasure hunter 10: 13
Total Treasure hunter 11: 20
Total Treasure hunter 12: 2
Total Treasure hunter 13: 2
Total Treasure hunter 14: 1

Total Treasure Hunter Procs: 112

Total attacks: 25,183
Avarage attacks per mob: 627.575
Average Treasure Hunter procs per mob: 2.8
Average total attacks to get a Treasure Hunter proc: 224.84
------------------------------------------------

The most important number in each of these sets is the final figure; IE the average total attacks to get a treasure hunter proc. This is the treasure hunter proc rate. It is the total number of attacks in the set, divided by the total number of treasure hunter procs. We have to understand that while there is some variance in any one individual mob kill, over the course of the entire testing this rate remained nearly identical.

With a worn treasure hunter + 14 set, my treasure hunter proc rate was 1 proc in every 225 swings, and with a worn treasure hunter + 8 value my average treasure hunter proc rate was one proc in every 231 swings. The percentile difference between the two is a mere 2.5% difference, and that can be accredited to variance in the sample size. This is a sample size of roughly 50,000 swings, divided into roughly 25,000 swings per set. When we're dealing with something that appears to occur roughly once every 225 or so times on average, that margin of error is well within statistical norms for a sample size like this.

The numbers I'm seeing also follow along with the correlation between the treasure hunter 14 data I colected when I was using sandung versus the data with double perfect taming saris. With the larger number of swings per mob (due to sandung sucking at dealing damage) I was getting roughly 250 more hits in per mob than I was with double perfect tamis. Consequently, the average treasure hunter proc per kill with sandung was almost spot on at 1 level higher than those kills with two tamings. There's close to a flawless correlation with the extra hits that sandung got in and the rate of treasure hunter proc aligning with the end values.

The summary of all this is that there truly appear to be no benefit to wearing more than TH +8 in equipment. If you wanna proc stuff, the most optimal way to go about it is with our relic gloves +3 and a perfect lucky egg.

200+ attacks to get a single TH proc? (I use crit hit rate +60% in gear to get much more frequent procs). Also, allegedly TH only procs on the mainhand swing, not offhand....However, I feel like Fighter's roll's Double Attack breaks this rule. (I know this is anecdotal, but still).

Thank you for doing this!

How did you capture the Data? I would like to replicate the experiment.

Kparser. It can be found in this thread here on page 5

https://www.ffxiah.com/forum/topic/16905/new-memloc-for-kparser/5/

Cerberus.Gillesjboulon made the post


You have to download the modified sources he posted and then merge them with the binary program (the bin).

Then you have to update the sql database by installing sql server compact 4.0. Microsoft changed the web address where the download is located when they incorporated service pack 1 so the link in the thread doesn't work anymore. The new location is here

https://blog.sqlauthority.com/2012/09/15/sql-server-download-microsoft-sql-server-compact-4-0-sp1/

Then finally when the program is running you have to go into Tools --> Options and update the memory offset address. The current memloc is 00621338.

Do all that and you've got yourself the best parser the game has ever had. Programmed by my old friend motenten, AKA Kinematics. I miss playing alongside him...

While I quite like seeing data on this, I don't think the methodology isolates the variable correctly, nor do I think the conclusion drawn based on the data provided is accurate.


Note that while you're looking at average total attacks to get a treasure hunter proc, you're not isolating for current treasure hunter level. We know full well that higher levels have a lower procrate, meaning that without isolating for that you create a potential situation where TH14 is being punished for upgrading(spending longer fighting at a lower rate) which will skew data.

Also keep in mind that while your average attacks to proc TH were lower by a small factor, your actual TH levels were notably higher, meaning more attacks were spent hitting a mob with an inherently lower procrate. This could be indicating that the additional levels of TH do help by more than your original data indicates.

Finally, consider that you're spending longer on the same mob if the higher TH set deals less damage, and additionally you have to consider that if TH can only proc on the first hit your higher TH set may be falsely presenting itself as better than it is due to having less multihit gear, and thus less attacks that cannot proc.

If you want the test to hold water, I would advise setting up an addon to log and a bot to farm, and collecting data on the basis of #/first hits vs upgrades at each given TH level. Using an automated setup to collect data will also enable a sample size worth assessing, this one is far too small to be conclusive.

A 3% difference is enough that you can't write it off entirely with a sample of that size, but given how many factors were not isolated it's not able to be used as evidence in either direction. It's also worth noting that it's possible additional TH helps when compared to active TH rate, and the level is capped based on difference. This could mean that additional TH does nothing while sitting at TH8, but helps at TH9+, TH10+, etc.

Jp test seems to show that critical hits doesnt have that much effect on TH proc if any.

JP test seems to have more data and conclusion seems to be TH above 8 doesnt help.

Should post JP test and methodology then. I have no reason to believe it does or doesn't help, but this test is certainly not anywhere near conclusive on it's own.

See, I've been telling people this for awhile, and people thought I was nuts for asking for data for their claim that it helps...

I do not think 40 mobs is not enough.

What appears to be a small difference in the Average total attacks to get a Treasure Hunter proc: 231.792 vs 224.84, resulted in a significantly skewed distribution curve of final TH outcomes on mobs.

With TH8 gear, TH13 nor TH14 were never achieved.

With TH14 gear, 2 TH13s, 2 TH14s. It could be the gear does not help increase the proc rate much or at all at TH8,TH9,TH10, but it helps more and more the higher the TH level.

This means the total number of additional TH procs will not be large. But they are significant because the additional 3-4 procs happen at TH13+

The limited data seems to support this - but would need a much larger sample set of total mobs.

Probably a stupid question, but do we know if it is even possible to hit TH13/14 with only TH8?

Nevermind that! Sticky thread discussion already answered with a "yes"

Seconded. The limited data seems to support additional TH rather than less because of the distribution leaning towards higher final proc

The higher final procs though are skewed from the data where individual swings per mob was higher. I'll try redoing the test sometime with a consistent TH 14 set throughout the entire test and see what happens.

Yes, you obviously want to see more data. Always.

The point is that, doubling the TH 8>14, barely, if at all, shows any real increase. One would reasonably expect if raising the TH from 8 to 14 that the increase in rate should be more significant than it was shown to be.

If an increase of nearly double the "strength" of the TH only produced a 2% increase, that's sufficient to say it does nothing.

It would be better to show the increase from TH3 to 4 to 5 to 8 to 14 and you could see the slope decrease as you hit 8+ , but that's a lot of effort for nothing. what's there is good enough.

This is exactly what I've been trying to point out the whole time. If wearing TH 14 had a significant amount of performance increase over TH 8 then it should be visible from just this much data. Keep in mind this is approximately seven hours worth of constant nonstop swinging. And the values I generated were nearly identical.

I'll rerun the test with a larger sample size, and this time I'll use actual buffs to expedite the process a bit. Red curry buns, sylvie, qultada, and ulima instead of purely healer trusts..... etc. But the takeaway is that there's no value in wearing more than TH 8. The benefit it provides.... if it exists at all... is so miniscule it may as well be nonexistant.

Silly question: by "wearing TH8" do you mean 5 in gear and the native traits? or 8 in actual gear? and further silly: what about the JP gift (if applicable?)

My thief has been mastered for years. I've had all 2100 job points with 500 stored that I can't spend since like.... 2017 now. And yes, for TH 8 I mean TH 5 in gear + native traits. Similarly, TH 14 is 11 in gear with native trait.

Thank you for the data and work you have done.
i was nearly going to disagree on your conclusion but then i re-read before making an *** of myself.


I do believed maybe some validity in the argument of distrnution mighh have had an unlucky effect.

Would it be possible to sort your data for just swing on a th8 mob and he proc rate?



but i agree with asura.eiryl
"but that's a lot of effort for nothing. what's there is good enough."

because proc rate x number of trying to proc = results in a higher outcome
The increase is form more swing not more proc rate

examples if i ship 5 trucks with 1 tone stone in each vs only one truck. i will still ship more tons of stones but the truck did not get a bigger load ability. the increase is from more trucks

How can we help?

Are we defining significant as at least 5%?

Should we all be helping generate data for a larger set and combine it at the end?

Came to say basically this.

You state that it has no effect but even your testing shows improvement. Sample size is way too small.

Glad you're digging into it though, it's great to have more info

Not quite.

Even if you account for the larger number of attacks with T14 (7%) more, the number of TH upgrades at TH14 gear is over 10% higher.

Yes it is a tiny difference - but we are talking about TH here - where SE loves these small percentage boosts.

Without more data it is hard to know.

Another point - this analysis is treating a TH upgrade from 8 to 9 the same as a TH upgrade from 13 to 14. Is it? I think a TH upgrade from 13 to 14 is harder to achieve and worth more when it does. But that is hard to quantify that here.

So I took a different approach.

I took the final TH outcome of each of the 40 mobs - and performed a lookup to the "ultra rare" TH values provided by SE.

https://www.bg-wiki.com/ffxi/Treasure_Hunter

This gives us the average TH throughput - which, in the end, is all that really counts.

For the TH8 gear set - it is .9587
For the TH14 gear set - it is .9925

This way of looking at the data is not perfect either ... because the more attacks arounds you have the higher the chances for an upgrade - and if the attack rounds are not exactly equal then a direct comparison cannot be made.

You also have the problem where once TH14 has been achieved, any attack round after that on the mob is diluting the data, because a TH upgrade cannot be achieved after TH14 - so if there are, for example, 5000 more attack rounds on a mob after TH14 happens, the attack round data is skewed. This could be solved by killing the mob immediately after TH14 happens, but I do not know if this is how the data was collected.

With all this analysis - I still have no idea if TH14 gear is better, but based on the small sample set - it could be.

@Melphina - Did you save the raw TH upgrade data? (not just the final TH outcomes)

If the question is 'is it practical to wear TH14', then yes you can reasonably conclude 'it is not practical to wear TH14' based off of this.

If the question is 'does additional TH increase proc rate further', I don't think you can. A relative 1% gain per extra point of TH is almost equally well shown(6 extra TH resulted in a 2.9992% gain in proc rate), and that's before we look into any of the previously mentioned complicating factors.

I realize 7 hours of testing is a lot of work, and I'm not trying to undermine that effort, but the better way to do it would have made an addon that logs your current equipment against your hits and mob's TH level. This could've been accomplished in a couple hours and would allow crowdsourcing to provide isolated data points that are both high in quality and volume. Things like critical hits and extra hits from attack round can also be measured. Kparser might be what you consider the best parser, but it's far from the best tool in this situation, given the extent of knowledge the community has about packets at this point.

If you're talking about how many hits in the log it takes for each upgrade, Kparser isn't able to record that. I just pulled an apex matamata and recorded the number of swings and subsequent TH level after the kill. I'll see about trying again with some actual buffs to approximate a more accurate in game scenario. The goal of this project is to determine the practical effects of wearing more TH than what is necessary. So perhaps things will change with lower swing counts per mob, but I'm not expecting a lot. We'll see.

I was just looking for a list of each TH upgrade that happened, and the TH level of that upgrade.

Thanks

lm

There's the issue of packet delay mixing up the order of things in the chat log though. It's been proven that when you're swinging extremely fast the order of events in the chatlog doesn't necessarily correlate to the order they occur on the server.

The order of packets correlates to the order things happened on the server, which is why I recommended using an addon. Attack rounds are sent as a single packet containing up to 8 swings(probably the real reason swings are capped at 8, rather than SE actually caring about a max swings/round). The actual TH proc has a visible animation which is flagged to a single attack within the packet, in addition to the log message. You can break down the attack packet and see if any trends emerge(such as TH always being listed with the first or last attack) then further break the data down by seeing if attack rounds with multiple swings have a proportionally higher chance of TH proc. This would answer the questions:
-Is it first swing only?
-Does it display on first swing only, but actually count all hits?

There's just a whole lot more you can do with a relatively thrown together addon than you can with kparser. Keep in mind that every single attack isn't it's own packet, despite windower and ashita nomenclature. The server queues them up and sends a bunch together in a single UDP packet, which preserves order. The chat log is less reliable because many events are coded to only show up in the chat log once their animation has completed, rather than as soon as the data is received.

Edit: 1000 edits later, think i covered everything

I performed another analysis of the data.

I want to make sure we take into account the weight of each TH upgrade.

Using the TH "ultra rare" column in bgwiki TH data, I have assigned a TH upgrade value to each TH level. (the difference between the % of the prior level to the next level)

TH9 = .1
TH10 = .1
TH11 = .1
TH12 = .15
TH13 = .15
TH14 = .2

Using this data we assign a value to every TH upgrade in both data sets - then we take an average over each set.

TH8 = 101 TH procs - total value of TH upgrade is 10.35
Attack rounds = 23,411
Average Treasure Hunter % boost per attack round: 0.0004421


TH14 = 112 TH procs - total value of TH upgrade is 11.7
Attack rounds = 25,183
Average Treasure Hunter % boost per attack round: 0.0004646

Wearing TH14 data results in a 5.089% boost in TH drop percent for the "ultra rare" drop class.

Based on the small data set - it seems to help.

Honestly dude, you've flipped back and forth 3x and you're using data that has no basis in anything besides your own arbitrary design. I don't see how that shows anything.