|
|
PacketFlow - For Ashita and Windower
 Asura.Saevel
Serveur: Asura
Game: FFXI
Posts: 9933
By Asura.Saevel 2022-06-02 21:29:21
So I have no idea about any of this stuff but does increasing bandwidth by 60% increase server cost for SE? Assuming enough people are using this.
That's ... not how this works. What it does is increase the effective interval that the client gets updates from the server, not the size of those updates. If the server doesn't have much data to give you, then this won't have any impact at all. If the server had too much data to give you, forcing it to discard some, then this will allow you to get more of that data. It's raising the effective data cap.
So no SE's bandwidth usage won't go up by 60%, probably won't even make a noticeable change when we take all the factors into effect.
 Asura.Bippin
Serveur: Asura
Game: FFXI
Posts: 1090
By Asura.Bippin 2022-06-02 21:33:54
So no SE's bandwidth usage won't go up by 60%, probably won't even make a noticeable change when we take all the factors into effect. Thank you this was something I was clearly not understanding
 Asura.Raelia
Serveur: Asura
Game: FFXI
Posts: 75
By Asura.Raelia 2022-06-02 21:51:49
Let's simplify and suppose worst-case XI has 50k players all logged in constantly. That's a whole 125MB/sec or about 1 gigabit of bandwidth. For reference, I can get that much throughput from Linode at $5/month. It's still 10TB/day though which would eat through the data allocation in about 144 minutes.
Another fun reference is that XIV is considered 'light' at 20MB/hour but XI is just under half that.
They more than likely either have their own fat connection and/or likely pay no more than $20 per TB anyway (egress is like $7 elsewhere, but Asia can be rough for hosting) so their raw data budget is maybe $6k USD fully rounded up and worst-case.
Seeing as those 50,000 players pay them a minimum of $650k USD a month, I think that counts as 'drop in the bucket' and probably isn't really monitored. I doubt it would even look like more than regular price fluctuations or whatever. Now consider that only a fraction of a fraction of players are going to be using Windower and using this plugin and even if someone looked really hard I don't think it'd stand out at all at the raw bandwidth level.
They'd have to look at player traffic specifically, and the assumption that they'd know what to look for to detect this goes hand-in-hand with the observation that they don't have the engineers to do anything about the problem it solves themselves.
I do wanna point out I've had some quirky logins with it activated though. That's another reason for the 'only for a little while after zoning' suggestion.
Bahamut.Shozokui
Serveur: Bahamut
Game: FFXI
Posts: 460
By Bahamut.Shozokui 2022-06-03 00:12:07
This is really cool. Thanks guys.
[+]
Serveur: Bahamut
Game: FFXI
Posts: 25
By Bahamut.Immortalknightx 2022-06-03 01:16:39
The best question yet. How do I get it?? I don't see a download link
By SimonSes 2022-06-03 01:28:26
Bahamut.Immortalknightx said: »The best question yet. How do I get it?? I don't see a download link
Built in to windower. Konami code on plugin tab to unlock few hidden plugins including this one.
[+]
By RadialArcana 2022-06-03 01:37:44
I think it needs to be kept in mind people are going to and are probably already abusing this right now. They will probably change it to be even faster.
If I wanted to have bots constantly spamming the auction house for cheap buyouts, I would use this to be able to spam faster.
 Asura.Wotasu
Serveur: Asura
Game: FFXI
Posts: 342
By Asura.Wotasu 2022-06-03 02:20:32
I think it needs to be kept in mind people are going to and are probably already abusing this right now. They will probably change it to be even faster.
If I wanted to have bots constantly spamming the auction house for cheap buyouts, I would use this to be able to spam faster. Good luck to them, SE can see very easy who's abusing Bots to buy faster from the AH. Even Rooks can see it.
By RadialArcana 2022-06-03 03:28:58
To reiterate the warning
It's not some old codger in the XI staff running the servers, it's the XIV staff. All operations are carried out by them, only development is carried out by the XI team.
If you wanna use this, you really do need to keep in mind you're risking your account. There is no grey area here, they can see you doing this and it's going to cost them extra money if enough people start using this.
I don't think this is worth it personally, and if it is just use it in the areas you're having problems and turn it right off after.
Aside from anything else if armies of bots start abusing this to buy>sell on the auction house you're going to get lumped in with them.
If you don't wanna use this, the information offered is good alone. You can turn a lot of things off to lessen the load, you can use static armor in the config, turn on chat filters, turn off party buffs etc
[+]
 Asura.Jinbe
Serveur: Asura
Game: FFXI
Posts: 115
By Asura.Jinbe 2022-06-03 04:11:51
been trying this for the last 2 days, and there is only one thing can be said, THANK YOU!
its working great and doing well. never had a smooth segment run before, looking forward to see how it do in dyna !!
and ppl who keep mentioning SE & ban, they can ban 90% of the player for many reason anyway, its not our fault that they dont want to fix there game by there own.
Keep the great work guys!
Shiva.Thorny
Serveur: Shiva
Game: FFXI
Posts: 2852
By Shiva.Thorny 2022-06-03 05:45:46
"...and this graph shows that our 10k monthly bandwith/server costs in june increased to 20k, we're investigating the causes but we believe it might be due to a third party addon." Alright, first off, you're not doubling bandwidth. The quoted 60% is not likely in many cases, either. It is a maximum, not a constant. To be actually adding 60% load you need to have both:
-A round trip time consistantly below 250ms
-Enough data waiting on the server to be sent
In reality, most people are going to be above 250ms round trip time and not even using the full capabilities of this plugin. When there's not enough data for the server to send, it's not mindlessly filling the buffer to bloat communications. So, while you're sitting around in your MH, idling in town, or waiting for your event to gather it's likely under 5% impact.
Next, you can't assume more than 40% of the playerbase will ever use this. The real number is probably considerably less, 10%? 5%? Let's go with 40 anyway for illustrative purposes.
If 40% of the playerbase uses this, they are in active content 10% of the time (2 hrs and 24 min a day.. more than most of you, i'm sure), and they are not the rest..
.4 * .1 * 1.6 = 0.064
.4 * .9 * 1.05 = 0.378
.6 * 1 = 0.6
You're likely looking at a sum bandwidth increase of under 5%. The real concern isn't that they're suddenly going to have some bandwidth bill. They're not going to saturate their link either, because they self host and it's in the same facilities as xiv(as RadialArcana likes to continually point out).
Hell Yes! and its not only bandwidth, its CPU cycles to present the data as well.. This is a slightly better complaint, but you're not increasing the number of actions you're sending out or doing anything that requires considerable calculation. Most of these packets are prepared either way, the biggest consequence is that there will be more movement update packets. Fortunately, the servers are already calculating mob positions every tick, and action distances are calculated at the time of action. The most likely scenario is that they're just copying in your new position slightly more frequency and sending out slightly more player/mob positions to you, both of which are cheap operations. It could be more considerable if they recalculate distances to every entity when you move, but it's still not likely to be a meaningful percentage of the server's cpu time. That doesn't mean it can't be the difference between maximum load and falling behind though, especially on Asura. It's a real possibility, just not one I consider particular likely.
If I wanted to have bots constantly spamming the auction house for cheap buyouts, I would use this to be able to spam faster. If you've ever had bots spamming the AH, you'd know that's one of the things they still ban for in a very short time period. If you bid faster than the current 7 seconds(?) the client takes for any sustained period, you'll be banned in a couple days. So, this would be pointless unless you're on a mule you plan to get banned, in which case you're paying SE the 10 bucks an account and covering your own bandwidth anyway.
They will probably change it to be even faster. There's really not much point in doing this. If you reduce it below your round trip time, you'll end up with all sorts of problems as things desync. If you were to try to binary edit the constants in this plugin, you still wouldn't be reducing it below your round trip time, so in most cases you're not actually going to change the speed(because, as stated earlier, you don't get the full reduction unless you have ping low enough to use it). If you know how to go in and do a more complicated edit, you probably could've written something like this yourself.
That said, why would you bother? Change load times from 2s to 1.8s? The goal is to reduce the threshold such that excess data can be fit in, and the limitation we set has been tested to adequately handle that in most real situations. Most of the people using it are not at 250ms round trip as is, and are still seeing significant improvement. Even if someone did edit it, I have to believe the majority of players will take our word for it and understand that the settings are the way they are for a reason; we've had numerous people asking for the ability to self-impose various limitations, while nobody asked for it to be made faster.
Shiva.Thorny
Serveur: Shiva
Game: FFXI
Posts: 2852
By Shiva.Thorny 2022-06-03 05:57:05
If you don't wanna use this, the information offered is good alone. You can turn a lot of things off to lessen the load, you can use static armor in the config, turn on chat filters, turn off party buffs etc As a side note, the only one of these things that actually does anything is using static armor. Turning off party buffs won't stop the data being sent, neither will filtering chat. When you're moving, the movement and gear change go into the same packets anyway.
While we're crying over this small indie company's bandwidth, I should point out that idling in Asura's port jeuno instead of your mog house for 8 hours a day is mathematically adding more bandwidth costs than using this would. Is it time to have a 'bazaar tax' paid in crysta?
By RadialArcana 2022-06-03 06:14:42
It's so dumb they have a way to turn off
/statusparty on/off
and they still send the data
I guess there are other things you could do though, turning off Unity for example and taking off Linkshell would probably have a decent impact maybe?
Shiva.Thorny
Serveur: Shiva
Game: FFXI
Posts: 2852
By Shiva.Thorny 2022-06-03 06:22:18
I guess there are other things you could do though, turning off Unity for example and taking off Linkshell would probably have a decent impact maybe?
There's really not a whole lot you can do. Reducing or stopping your own gear swaps would be the biggest impact, because every single piece needs to be sent, and most of the time when it saturates it's on or slightly after a return handling your 30-32 swaps from precast/midcast. At 8 bytes each, that's 240 bytes or more for doing a heavy pre/mid swap. Your own appearance gets sent once every time you swap(but not once per piece), that's 26 bytes you can block by using lockstyle. Other player update packets are 106 bytes each, you can potentially reduce the number of those being sent if others use lockstyle, but they'll still need to go out whenever the player's hp or position changes. Most swaps will change max hp enough to alter your HP%, so other people's lockstyle doesn't help a whole lot. Having your whole group stop equipment swapping would almost certainly have a huge impact, so ironically, you have gearswap and similar as a big contributing factor to these, especially when combined with react and other things to be blinking more frequently than you use actions.
Chat messages are not likely to take as much data as even one player update, so they're a drop in the bucket. They aren't sent constantly either, unless you have a particularly loud and active linkshell, so I'd say it's barely worth consideration.
 Asura.Sechs
Serveur: Asura
Game: FFXI
Posts: 10136
By Asura.Sechs 2022-06-03 06:27:15
I want to thank Byrth and Thorny for resolving almost completely what has been in my opinion the biggest technical issue within FFXI over the last few years.
Personally I think I'm going to be using this. Wether I'm going to keep it constantly on or just when I enter an instance I don't know, haven't made up my mind yet, but I'm willing to risk it.
I personally think the risk is minimal and I agree with the reasons Thorny/Byrth have been kind enough to share with us in an open, transparent and quite logical way.
It may sound cynical for me to say but personally, given that this "solution" has been found, I am no longer willing to play the game without this solving for this problem that has been a real bane for my personal enjoyment of the game over the last years.
I seriously doubt this will end up with a ban for my account, but even if that were to happen, I will ultimately accept the sad, sad consequence.
Just my two cents, of course.
I'm happy everyone will be free to get to their own personal stance on this matter.
Shiva.Thorny
Serveur: Shiva
Game: FFXI
Posts: 2852
By Shiva.Thorny 2022-06-03 06:29:39
I do wanna point out I've had some quirky logins with it activated though. That's another reason for the 'only for a little while after zoning' suggestion. Missed this, sorry for so many posts back to back. Both Windower and Ashita versions deactivate when you zone out, and resume on zone-in. They make the process of zoning feel faster because of the data population after zonein, but are not active during the zone transition itself.
In the Ashita version, it starts off inactive if you aren't already logged in, but it is possible that isn't the case for Windower and it's defaulting to on for first login. It's a relatively easy change to make, and I will bring it up to Byrth to verify.
[+]
Lakshmi.Byrth
VIP
Serveur: Lakshmi
Game: FFXI
Posts: 6194
By Lakshmi.Byrth 2022-06-03 06:47:26
In the Ashita version, it starts off inactive if you aren't already logged in, but it is possible that isn't the case for Windower and it's defaulting to on for first login. It's a relatively easy change to make, and I will bring it up to Byrth to verify.
Fixed this in the Windower version and now it is working as intended.
By RadialArcana 2022-06-03 08:00:46
Does this increase load time of wardrobes or is that already capped out.
Shiva.Thorny
Serveur: Shiva
Game: FFXI
Posts: 2852
By Shiva.Thorny 2022-06-03 08:05:18
Does this increase load time of wardrobes or is that already capped out.
It makes wardrobes load faster, yes, and typically by close to the full 60% because you will be saturated while zoning. Note that this increases peak bandwidth, but not sum bandwidth, because the data would've been sent at some point anyway.
By GetHelpNerd 2022-06-03 08:21:50
Does this increase load time of wardrobes or is that already capped out. Everything you have asked so far was already covered in this thread. This question and the 10 others you've brought up.
Just absolutely astounding.
You also managed to tie this whole subject back to FF14 somehow.
There are no words for how few cycles your brain uses. It would greatly benefit from this plugin if we could just inject it into you.
It's not some old codger in the XI staff running the servers, it's the XIV staff. All operations are carried out by them, only development is carried out by the XI team.
By RadialArcana 2022-06-03 09:14:50
Does this increase load time of wardrobes or is that already capped out. Everything you have asked so far was already covered in this thread. This question and the 10 others you've brought up.
Just absolutely astounding.
You also managed to tie this whole subject back to FF14 somehow.
There are no words for how few cycles your brain uses. It would greatly benefit from this plugin if we could just inject it into you.
It's not some old codger in the XI staff running the servers, it's the XIV staff. All operations are carried out by them, only development is carried out by the XI team.
lies
By GetHelpNerd 2022-06-03 09:39:21
Fenrir.Jinxs
Serveur: Fenrir
Game: FFXI
Posts: 541
By Fenrir.Jinxs 2022-06-03 09:46:03
Anyone got a comparison video?
By Aricomfy 2022-06-03 10:13:36
When using 60 fps mode in the config addon there are very apparent slow downs in some areas such as Western Adoulin. Would PacketFlow reduce this issue for 60 fps mode? I wouldn't think so but just to make sure, I figured I'd ask.
Serveur: Asura
Game: FFXI
Posts: 116
By Asura.Neviskio 2022-06-03 10:26:04
I appreciate the clarification Thorny as that makes it a bit more in perspective, and yeah my numbers were just cause the 1 and 2 are near on the KB and it makes it more impactful as a point, wasn't implying it doubles or that I know any of their numbers. Just wanted to make sure there was indeed a clarification as the implication of increasing cost would make someone notice this more than any other addon. Now if the increase scale is minimal that it disappears in the normal fluctuations then they might never care. Still the least it affects their bottom line, the safer it gets.
While we're crying over this small indie company's bandwidth, I should point out that idling in Asura's port jeuno instead of your mog house for 8 hours a day is mathematically adding more bandwidth costs than using this would. Is it time to have a 'bazaar tax' paid in crysta?
But kinda wanted to point out on the above quote, there's a bazaar tax, it's called "monthly subscription" and it does cover the bandwith cost of being online on the server and using the normal services... y'know? <.<. I get what you mean but keep in mind we're already paying for a certain amount of service they expect, as barebones as some may feel... and the point of dealing with square enix is not insignificant considering their *** with NFT/greediness so it makes it even more important to make sure the bottom line isn't affected negatively or they will shut it down even more, or try to grab even more money in other ways. Generally these companies account for everything quite aggressively and even small fluctuations can affect the decision making.
In my personal experience anyone above the dev team only cares about cost/profits so even a 1k a month will be noticed, if that's enough to take action I don't know, but I wouldn't be surprised myself so it's worth making sure it helps the most while affecting them the least, in my mind.
Lakshmi.Byrth
VIP
Serveur: Lakshmi
Game: FFXI
Posts: 6194
By Lakshmi.Byrth 2022-06-03 10:43:16
When using 60 fps mode in the config addon there are very apparent slow downs in some areas such as Western Adoulin. Would PacketFlow reduce this issue for 60 fps mode? I wouldn't think so but just to make sure, I figured I'd ask.
Problems that only occur when you increase your FPS will not be solved by this.
Anyone got a comparison video?
It's going to be hard to make a meaningful comparison video here of anything but the time it takes to load a zone.
By RadialArcana 2022-06-03 10:50:32
When using 60 fps mode in the config addon there are very apparent slow downs in some areas such as Western Adoulin. Would PacketFlow reduce this issue for 60 fps mode? I wouldn't think so but just to make sure, I figured I'd ask.
Would you be interested in trying a mod of that zone to see if it fixes your issue?
Someone asked me about this years ago and I made a mod for it, I can give you a link to try if you want.
Shiva.Thorny
Serveur: Shiva
Game: FFXI
Posts: 2852
By Shiva.Thorny 2022-06-03 11:44:28
I get what you mean but keep in mind we're already paying for a certain amount of service they expect My point is that your personal impact as a player who is otherwise thoughtful and idles in mog house or logs off overnight is greater than the impact of using this plugin. Obviously some people will both use the plugin and afk in a high traffic area. But, if you're concerned about your personal footprint on SE's servers.. there are a lot of ways to reduce it, and I've never heard anyone concerned that you're using too much bandwidth by leaving your character on in town 24/7.
so it's worth making sure it helps the most while affecting them the least, in my mind.
Yea, this is definitely a sentiment that's been echoed throughout the thread. We have a few more settings being discussed and implemented that will allow users to reduce their own impact even more while still preserving the benefits they find most important. Details will follow once they're live.
Asura.Saevel
Serveur: Asura
Game: FFXI
Posts: 9933
By Asura.Saevel 2022-06-03 11:55:14
It's kinda hard to talk folks seriously talking about impact when they use gearswap for pre/mid/after along with all the other super awesome tools in windower. This plugin really doesn't have much impact at all, unless your in a place where your client would be bombarded with actions, like Dynamis D.
[+]
By Aricomfy 2022-06-03 12:08:34
Would you be interested in trying a mod of that zone to see if it fixes your issue?
Someone asked me about this years ago and I made a mod for it, I can give you a link to try if you want.
I was lurking the thread when this issue was brought up and I believe the root of the problem was the smoke from the chimneys coming off of almost every building or something that was lagging the game more than normal. Was it your mod that "removed" the smoke with a texture edit?
The slow motion effect with 60 fps mode happens in other circumstances with above average stress which is why I thought PacketFlow might actually help relieve this problem.
Now announcing, PacketFlow for both Ashita and Windower!
Players have been having persistent issues with "lag" causing messages to drop in instances for years. In reality, this "lag" can be decomposed into two separate problems:
1. Chunk prioritization is messed up in instances
2. Instances generate a lot of chunks
PacketFlow attempts to address the second problem.
For a brief overview of FFXI network dynamics and terms, I recommend reading the spoiler: FFXI communicates between the server and client using UDP packets that have a max size around 2.5KB. Those packets are composed of multiple "chunks" of FFXI data, carrying information to either tell the server what you're doing (client -> server) or for the server to update you on the state of the game (server -> client). People frequently use the term "packet" to refer to chunks, but I'm going to keep the terms separate here because it is mechanistically important.
Examples:
When my FFXI client displays a message like "Byrth gains the effects of Phalanx" in the chat log, it knows to do so because it received a chunk telling it to display a gain buff message for my character within a larger UDP packet that may also contain a lot of other information. When you see my character run around, it is because my FFXI client sent the server a message telling it my new position (one chunk in a client->server UDP packet), and the server relayed my new position to your FFXI client (another single chunk in a server->client UDP packet).
In non-instanced zones, SE appears to use a prioritization system to make sure that your client receives more important chunks. If you pull all of the monsters in Korroloka Tunnel at once, for instance, and cast Phalanx on yourself, you will always see the "gain the effect of Phalanx" message. More generally, chunks containing information involving your character are prioritized over more contextual chunks (like monsters moving around). Most of the time, this prioritization system does not matter because FFXI is sending you less than 2.5KB of data and so everything makes it into the UDP packet. However, sometimes you're in a very active environment and there's more than 2.5KB of data to send you.
You can experience this when by pull multiple monsters. If 5 monsters are chasing you, they typically move pretty reasonably. If 20 are chasing you, some monster start to jump between locations because the chunk that updates those monsters' positions have low priority and do not fit in the packet. However, even with 20 chasing you the messages about their attacks are generally included (and fill up your chat log). This prioritization system lets FFXI get away with a pretty low bandwidth requirement, and the fact that most of us have never really thought about it indicates that it works pretty well.
However, in instances the prioritization system is at best nonfunctional or worse is functioning in the opposite way from normal zones. You will frequently miss messages about yourself in a Dynamis - D instance (buff gain, action completed, etc.), that you would never miss in a vanilla Dynamis zone. This is all server-side, so we can only guess at the specifics of the packet prioritization system and how it's messed up in instances, but it's clearly not working as it was designed to 2 decades ago.
What are solutions here?
1. Minimize the number of chunks you are sent: fight one monster at a time, turn off party buff information, don't use GearSwap, etc. Anything that decreases the number of chunks you are sent for FFXI is going to help you avoid missing chunks, because prioritization doesn't matter unless the packet is full.
2. Increase the server -> client bandwidth.
Option #1 sucks and is very limiting, so option #2 is the interesting one.
Bandwidth is more or less max packet size * frequency. For the FFXI server -> client connection, this is (2.5 kb / 1 packet) * ( 1 packet / 0.4s) = 6.25 kb/s most of the time. We cannot change either of those parameters directly because they are part of the server software.
However, years ago I noticed that the server responds to any UDP packet it is sent with another UDP packet. Thus, we can indirectly control the incoming UDP packet frequency by adjusting the outgoing packet frequency. This idea came to nothing for the better part of a decade mostly because I had no idea how to trigger the FFXI client to send UDP packets more frequently.
I was reminded of it earlier this year when KnugenJulian started posting in the Windower Discord asking if anyone had a fix for Odyssey lag. I brought up my purely theoretical fix, and Thorny made it real by connecting it with atom0s' UDP packet handler reverse engineering.
Thus, there are now plugins (both named Packetflow) for Ashita and Windower that decrease that time-per-packet from 400ms to 250ms, effectively increasing bandwidth by 60%. They do not completely solve the problem of dropped chunks in instances, but they do make it better. Additionally, they make loading zones quite a bit faster. I trained 20 Orcs in Dynamis D - Sandy on my RUN, running them 20' back and forth so they'd keep moving with two WHMs to keep me alive. The only purpose of this was extra packet generation (many movement and attack chunks from the monsters).
I also made a party of 6 (brd x1, rdm x5). Each RDM cast 6 barspells in rotation 20x (120 casts total). They precast naked and midcast in full 16 slots to generate equipset chunks. I set up a Lua to log cast completions and loaded it on two of the RDMs, one with PacketFlow and one without.
600 casts actually occurred, and:
* the character with packetflow saw 588, and all 12 missed were their own. (10% missed of their own, 0% of others)
* the character without packetflow saw 517, and 36 of those missing 83 were their own. (30% missed of their own, 10% of others)
As with all 3P tools, but perhaps even more here, you risk your account when you use this plugin. Packetflow is unambiguously detectable and increases the outgoing bandwidth costs for FFXI. Thus, for Windower it is in the Konami code section of the launcher.
As far as due diligence, we actually more or less finished these plugins in mid-April and I have been using the Windower one on my main and mule for the last month ~24/7. I'm not banned yet. We don't have a way to test it better than that and that's all of the testing that has been done. Caveat Emptor!
Attempts have been made to reach out to SE about these problems and to get a temperature read on the proposed solution:
* 2016 AMA - Bandwidth - basically "it would be too hard to fix"
* 2022 AMA - Prioritization
* 2022 AMA - Bandwidth
We waited a week for SE to post their hypothetical 2022 AMA forum follow-up where they answer those questions, but no dice.
Credit to:
* Thorny for the implementation and empirical testing
* atom0s for the reverse engineering that made it possible
* KnugenJulian for starting the conversation that revived the concept
|
|
