I just tested my Bookworm's Cape with "Helix eff. dur. +20" and it appears to be a percentage duration increase. It added 46 seconds to my helix duration (228 * 1.2 = 273.6). So it's essentially a 20% damage boost. So a Lugh's Cape would have to do 20% more initial damage to beat it. I don't think that's possible ...?
If your test turns out to be true it's 4 more tocks, which is a HUGE increase in damage.
But the problem still stands: will the monster last enough for the 4 tocks to take effect? Does the mob have abilities that allow him to erase the Helix effect?
How much damage does Lugh add over Bookworm when you're far from reaching the 10k threshold? I imagine a lot but clearly it can't be that much.
Some examples:
If your helix without Lugh hits for 3k, Bookworm would add 12k more damage.
If Lugh allows you to reach 4k that would be 24.000 more damage (24 tocks with Dark Arts and capped JPs. Even more if you use Tabula Rasa of course).
If Lugh allows you to reach 3,5k that would be 12k, which is the same damage added by bookworm.
Lugh duration is ~4', Bookworm (capped) duration would be ~4'48"
X = BookwormDMG *4 - (LughDMG-BookDMG)*24
If X > 0 then it's beneficial to use Lugh in place of Bookworm, the higher X is above 0, the more beneficial. All of this while keeping in mind that BookDMG and LughDMG need to be < 10k because beyond that there's no point.
So tl;dr
Pro players will need to have a manual trigger to turn on and off and decide wether they want to cast with Bookworm or Lugh.
If you only want to pick one and call a compromise, I don't really know what to suggest, but probably between the two I'd go for Lugh.