agreed, heatmaps with logarithmic cell intensity are the way to go for massive datasets in things like 10,000-series line charts and scatter plots. you can generally drill downward from these, as needed.
uPlot maintainer here. this looks interesting, i'll do a deeper dive soon :)
some notes from a very brief look at the 1M demo:
- sampling has a risk of eliminating important peaks, uPlot does not do it, so for apples-to-apples perf comparison you have to turn that off. see https://github.com/leeoniya/uPlot/pull/1025 for more details on the drawbacks of LTTB
- when doing nothing / idle, there is significant cpu being used, while canvas-based solutions will use zero cpu when the chart is not actively being updated (with new data or scale limits). i think this can probably be resolved in the WebGPU case with some additional code that pauses the updates.
- creating multiple charts on the same page with GL (e.g. dashboard) has historically been limited by the fact that Chrome is capped at 16 active GL contexts that can be acquired simultaneously. Plotly finally worked around this by using https://github.com/greggman/virtual-webgl
> data: [[0, 1], [1, 3], [2, 2]]
this data format, unfortunately, necessitates the allocation of millions of tiny arrays. i would suggest switching to a columnar data layout.
Really appreciate you taking the time to look, Leon - uPlot has been a huge inspiration for proving that browser charts don't have to be slow.
Both points are fair:
1. LTTB peak elimination - you're right, and that PR is a great reference. For the 1M demo specifically, sampling is on by default to show the "it doesn't choke" story. Users can set sampling: 'none' for apples-to-apples comparison. I should probably add a toggle in the demo UI to make that clearer.
2. Idle CPU - good catch. Right now the render loop is probably ticking even when static. That's fixable - should be straightforward to only render on data change or interaction. Will look into it.
Would love your deeper dive feedback when you get to it. Always more to learn from someone who's thought about this problem as much as you have.
Blind sampling like this makes it useless for real-world statistics of the kind your users care about.
And column-oriented data is a must. Look at Rlang's data frames, pandas, polars, numpy, sql, and even Fortran's matrix layout.
Also need specialized expicitly targetable support for Float32Array and Float64Array. Both API and ABI are necessary if you want to displace incumbents.
There is huge demand for a good web implementation. This is what it takes.
I once had to deal with many million data points for an application. I ended up mip-mapping them client-side.
But regarding sampling, if it's a line chart, you can sample adaptively by checking whether the next point makes a meaningfully visible difference measured in pixels compared to its neighbours. When you tune it correctly, you can drop most points without the difference being noticeable.
I didn't find any else doing that at the time, and some people seemed to have trouble accepting it as a viable solution, but if you think about it, it doesn't actually make sense to plot say 1 million points in a line chart 1000 pixels wide. On average that would make 1000 points per pixel.
We routinely face this in the audio world when drawing waveforms. You typically have on the order of 10-100k samples per second, durations of 10s-1000s of seconds, and pixel widths of on the order of 1-10k pixels.
Bresenham's is one algorithm historically used to downsample the data, but a lot of contemporary audio software doesn't use that. In Ardour (a cross-platform, libre, open source DAW), we actually compute and store min/max-per-N-samples and use that for plotting (and as the basis for further downsampling.
> In Ardour (a cross-platform, libre, open source DAW), we actually compute and store min/max-per-N-samples and use that for plotting (and as the basis for further downsampling.
I discovered flot during my academic research career circa 2008 and it saved my ass more times than I can count. I just wanted to say thank you for that. I wouldn't be where I am today without your help :)
> But regarding sampling, if it's a line chart, you can sample adaptively by checking whether the next point makes a meaningfully visible difference measured in pixels compared to its neighbours.
uPlot basically does this (see sibling comment), so hopefully that's some validation for you :)
Is there any techniques using wavelet decomposition to decimate the high frequency component while retaining peaks? I feel like that's a more principled approach than sampling but I haven't seen any literature on it describing the specific techniques (unless the idea is fundamentally unsound which is not obvious to me).
Interesting idea - I haven't explored wavelet-based approaches but the intuition makes sense: decompose into frequency bands, keep the low-frequency trend, and selectively preserve high-frequency peaks that exceed some threshold.
My concern would be computational cost for real-time/streaming use cases. LTTB is O(n) and pretty cache-friendly. Wavelet transforms are more expensive, though maybe a GPU compute shader could make it viable.
The other question is whether it's "visually correct" for charting specifically. LTTB optimizes for preserving the visual shape of the line at a given resolution. Wavelet decomposition optimizes for signal reconstruction - not quite the same goal.
That said, I'd be curious to experiment. Do you have any papers or implementations in mind? Would make for an interesting alternative sampling mode.
I don't. I just remember watching a presentation on it and it always struck me that wavelets are an incredibly powerful and underutilized technique for data reduction while preserving quality in a quantifiable and mathematically justifiable way.
I don't have any papers in mind, but I do think that the critique around visual shape vs signal reconstruction may not be accurate given that wavelets are starting to see a lot of adoption in the visual space (at least JPEG2000 is the leading edge in that field). Might also be interesting to use DCT as well. I think these will perform better than LTTB (of course the compute cost is higher but there's also HW acceleration for some of these or will be over time).
No, FFT is perfectly information preserving by definition. Thats why there’s an inverse FFT operation that restores the original signal without any loss (well, modulo accumulated floating point error when working in the discrete instead of symbolic space).
> creating multiple charts on the same page with GL (e.g. dashboard) has historically been limited by the fact that Chrome is capped at 16 active GL contexts that can be acquired simultaneously. Plotly finally worked around this by using https://github.com/greggman/virtual-webgl
Sometimes I like to ponder on the immense amount of engineering effort expended on working around browser limitations.
What I did in a few projects to plot aggregated (resampled) data without loosing peaks was to plot it over an area chart representing the min-max values before aggregating (resampling). It worked pretty well.
Both are useful. With the y-axis staying the same there is a stable point of reference. Then you can see how sub-samples behave relative to your whole sample.
related: i had to jump through the Date hoops recently (again) when rewriting uPlot's DST and timezone handling, but i'm pretty happy with the result that avoided much complexity, performance loss, and adding a dependency: https://github.com/leeoniya/uPlot/pull/1072
> An enduring myth about the Moon is that it doesn't rotate. While it's true that the Moon keeps the same face to us, this only happens because the Moon rotates at the same rate as its orbital motion, a special case of tidal locking called synchronous rotation.
My colleagues once spent a good hour trying to explain this fact to me and I still really struggle to accept it. I can see that the moon is rotating on its own axis from the point of view of a space that is external to the system it forms with the earth. But then isn’t everything on earth rotating about its own axis with respect to that external space? It seems arbitrary to isolate the moon from all this other stuff and make a special case of it…
1. Unlike position and velocity, which are relative (there is no given "origin" for them, no way to say where a thing is or how fast it's moving except relative to other things), rotation is absolute. A thing is either rotating or not, regardless of its relation to other things. Objects that rotate "experience (centrifugal) forces as a result" or "require (centripetal) forces to hold them together" depending on how you choose to describe it. This is detectable: hook two weights together with a newton-meter in space and the newton-meter will read non-zero when the assemblage is rotating, zero when not. The reading tells you how fast it is rotating regardless of any external reference point. (An equivalent device to detect position or velocity is not possible, but it is for acceleration.)
2. Yes, everything "at rest" on earth is in fact rotating at the rate the earth rotates. If you stand on the equator at midday and do not rotate you will be standing on your head at midnight.
>no way to say where a thing is except relative to other things
This is always true. The origin is just a thing that other things are relative to. It's just as possible to define an origin in the real world as it is on a piece of graph paper.
Thanks for this explanation. If I understand correctly then, the moon requires some centripetal force in order not to dissipate due to its rotation whereas e.g. my head or the Eiffel Tower do not because they are not subject to absolute rotation.
Indeed. The Eiffel tower and your head do both have some (extremely small) centripetal force compensating for their rotation along with the earth.
(You can break that down in different ways, i.e. use various choices of generalised coordinates to describe it, so exactly what constitutes "centripetal", "centrifugal", "gravitational", "tidal", etc. forces depends on that. I'm being pretty vague in how I decribe it. Regardless, rotation is absolute, or in other words the equations of physics take a different form in a rotating frame of reference than in a non-rotating one.)
Thanks for the clarification I completely mistook what you were saying. This is the fascinating bit for me then, that what’s happening with the moon’s rotation is also happening with everything else
> But then isn’t everything on earth rotating about its own axis with respect to that external space?
From the point of view of the moon, for the purposes of action due to gravity, anything on Earth is essentially part of the Earth, not an entity that is massive enough to be considered separately. The aggregate centre of mass is what counts. Similar for the Sun looking at the Earth/Moon system: from that PoV Earth+Moon is it a single mass with a centre somewhere between the two major masses that form it.
If the Moon where sufficiently consistent in its shape and density, it could rotate freely in any direction while orbiting the Earth, that fact that it is more dense on one side means that it is more energy efficient for it to spin in step with its orbit such that the dense side keeps facing us. If something massive hit the moon (let's assume this somehow happens without significantly affecting its orbit or causing significant problems for Earth too!) it might push the rotation off for a bit, but it would slowly be pulled back into sync. If something sufficiently massive simply landed on the moon, that would affect the mass distribution and the exact face that points at us would slowly change to reach a new equilibrium.
Pick the sun as reference: the moon rotates. Pick the earth as reference: the moon rotates. Stand on the moon and pick any star as reference: the moon rotates.
Yes but from the point of view of the earth the moon does not appear to be rotating around its own axis since it is tidally locked. In that sense it’s confusing to me to distinguish it from everything else on earth but the comment above about centripetal force clarifies this for me I think
> If I ever felt I was too old to have kids I’d probably have no choice but to kill myself
sperm quantity and quality decreases with age. studies exist that suggest higher risk of autism when father's age >= 45.
you're not too old, but you should probably test & freeze some good sperm before you might actually be too old by the time you find the right person. this way you won't ever feel like you're too old to have kids. then the question becomes more of "how long and in what manner will my remaining health allow me to enjoy them"
> Did we choose the age at which we would have children? What does it mean to choose?
we planned to have a kid by our early thirties. she specifically wanted one by 30. we were both healthy, financially stable with solid careers.
then came multiple miscarriages, 10 years of background/foreground stress, and IVF. now we finally have two healthy ones. i think daily about those 10 years we've lost to spend with our kids while still younger and able to do activities that i still enjoy like snowboarding, mountain biking, etc. thankfully i'll still be able to do some of it, but man, it has been rough. my awesome father-in-law died of cancer 9mo before his grandson's birth; the only thing he ever knew were our ongoing struggles :(
something else that happens is that all your same-aged friends with kids...they have different lives now. you can't talk to them about the same child struggles / tips in real time, the kids don't go to school together or know the same people; they're a generation apart. it becomes an isolating event when the delay is long enough.
despite all that, when i think of where i was financially then and now (and what i did in those 10 years to get from there to here that would not have happened otherwise), and that if i had a kid 10 years ago it would be a different [probably worse] kid instead of the adorable 2.5yo that runs to me each morning now, i feel a lot better.
my advice would not necessarily be to start earlier, but if you've decided to procreate and are consciously deferring it until the "right" time, just expect the really, really unexpected.
For decades the discussion in schools have been around "this is how you avoid unwanted pregnancy", safe sex and all of that.
With our(northern Europe) crashing fertility rate there's now also discussions about adding on "when the woman is 25 this happens and you're this likely to get pregnant, at 30 it's like this...", just so that people can plan and try for the family they want. If one wants 3 kids and don't want IVF you should apparently start around when the woman is 25-28 or something like that?
This is where free daycare, and support from the government helps.
(And yes of course it's not "free", it is paid for from taxes, people are so smart to point that out.)
Different countries have different incentives, but I was really pleased with the setup in Finland when we had our child. A free box of first-clothes, daycare from 1-5 years old cheap enough that it was almost free, and preschool at 6 before schooling started at 7.
Lots of minor perks, such as free transport on busses, trams, etc, if you were pushing a stroller, and so on.
Plenty of people have kids before then and they work out fine. I'm not saying that if you're truly destitute it's a good idea to have kids, but the only people I hear complaining about not being financially ready for kids are those who are objectively well off.
> my awesome father-in-law died of cancer 9mo before his grandson's birth
The death of the people I loved growing up is my biggest regret for leaving having children until my mid-thirties.
We have friends who got on with it early and their kids are great. The parents didn't have as much money or lived experience, but were fitter, more energetic, and now their kids are teenagers and they're able to focus on life. When our youngest is there, we'll be focussing on retirement.
It's impossible to know, I know, but every year deferred is another year less with a child that will probably love you, a love you will value above practically anything else.
the big irony, of course, is that i'm much more comfortable with China surveilling me than the US, since the latter can throw me in jail, seize my assets, and ruin my family's life, while the former cannot.
> it amazes me how often I see this kind of anti-democratic institition sentiment.
leeoniya didn't say anything about democracy. The practical reality is that regardless of what forms of government are involved, whichever government has the ability to arrest you is the government which is the greatest threat in your day-to-day life.
> government has the ability to arrest you is the government which is the greatest threat in your day-to-day life
Assuming every government is the same, which I'm not so sure about. I rather be arrested by the German government than the US government, mainly because I don't want to disappear to black site and be made to disappear for years while I'm t̶o̶r̶t̶u̶r̶e̶d̶ receiving enhanced discussion techniques. At least I know I'll be treated relatively OK by Germany, while my fear is pretty much the opposite from a lot of other governments out there.
Wrong. The American government is much better than the Russian government, but the Russian government cannot arrest me while the American government can, therefore the American government is a much more serious threat to me than the Russian government. No equivalence between the two governments is assumed or implied.
"The government that has the ability to arrest you" is the one that controls the police on the street you live on. Not some abstract commentary on which government is best at arresting people.
I've already placed my bets that current president will be the first to serve at least three terms since the two-term limit was introduced. Judging by what's happening, seems like a safer and safer bet every day.
I think the most likely reason that won't happen is some sort of cardiovascular failure (heart attack or stroke), not because anyone will actually stop the Republicans otherwise trying. Conservatives want a monarchy.
Shitty bet tbqh, but it's your money. Trump promises his supporters much but delivers very little. If J6 is the sort of insurrection his base can muster, there's no chance in hell of him getting another term.
He has said that he cannot do more than two terms, but also there are ways to do more terms. Then he said it's too early to think about, then that he is joking, then that he wasn't joking, then that he isn't looking into it, but that they're "probably entitled to another four after that" (https://www.cbsnews.com/news/what-trump-has-said-about-pursu...), whatever the fuck that means.
Ultimately, I don't think it matters much what he says or has said, he won't clearly say what he/they are planning, obviously.
> Ultimately, I don't think it matters much what he says or has said, he won't clearly say what he/they are planning, obviously.
Honestly they're pretty open about their plans. They laid most of them out in Project 2025. They just sometimes carry out those plans while also denying that they are following the playbook. Trump in particular will be surprisingly candid about what he's doing in between bouts of lies and denials.
Like he said he didn't know anything about project 2025?
Steve Bannon is the one working on this, has said they have a plan to do it. Trump himself seems to believe that if the country is at war elections are postponed because that is how it works in Ukraine. Ergo Venezuela.
It’s not anti-democratic, it’s simply a matter of exposure. China can WANT to do whatever they want to me, but I have no assets in China, no trade in China, and neither me nor anyone close to me will ever go to China. So it simply matters a lot less what China has on me than the country where I have friends, loved ones, financial assets, property, and frequently visit.
Generally I'd agree. The threats here are larger. That said China isn't powerless to hurt you either. I haven't seen much of it happening, but in theory China could blackmail you. They can manipulate and influence you and your children through social media and advertising, even encouraging kids to harm themselves/others.
They can also fill the products they make for us with heavy metals and other poisons while building them to break draining our finances and filling our country with trash. The worst thing they could do though is just stop producing crap for us entirely since we're basically dependent on them for just about everything.
And the united states can also do those things. We’ve been fighting against the hormone-filled milk for decades, and half of the ingredients are banned by smarter countries, but more than half of our food is still imported poison.
But none of that has to do with who is surveilling me online.
Yes the US is a democracy, but a lot of our systems suck ass and are also close in proximity. You DO NOT want to get into legal trouble in the US. Our justice system is beyond fucked. If there's one way to permanently ruin your life in the US, it's getting into legal trouble. You're better off smoking crack cocaine, that's probably healthier for your livelihood.
I don't know about China's legal system, but even assuming it's more fucked, it's all the way over there. Not here.
The main trouble with Flock and companies like them is that they attach to our broken systems like a tumor. If the system fails, which it often does, these accelerate it and make it worse. If you get falsely accused of something or piss off the wrong PD, this shit can ruin your life. Permanently and expeditiously.
Even if you are the most Moral Orel you should be skeptical of these crime reduction claims. They don't just beat down crime, they beat down regular people, too. And if you ask them, they don't know the difference.
> I don't know about China's legal system, but even assuming it's more fucked, it's all the way over there. Not here.
You're saying that the US legal system is extremely bad, shouldn't the assumption be that other countries have it better? I don't know much about either country's legal systems, but I do know that if I feel like my country is extremely bad at something, other countries probably do it better, at least that what I'll assume until I see evidence of something else.
I don't see how it matters how other countries rank vs the one a person lives in. Even if Canada's legal system is better than the US, you can't choose to subject yourself to the Canadian legal system without extricating yourself from the US first.
Maybe, I mostly gave that disclaimer to say that it actually doesn't matter much. Even if it's worse, that's still better, because it's over there.
But yes, generally, I assume virtually every developed country (and some of the kind of developed countries) have a more just and competent legal system than the US.
The US is an interesting beast, because when you compare it to the entire world on a bunch of stuff, it doesn't seem so bad. But when you compare to countries that have, like, clean running water, then it really falls flat in a lot of ways. This allows apologists to basically justify anything the US does, because somebody, somewhere, is doing it much worse. Hey guys, look at Uganda, they're genociding gay people!
Not being an expert in every single country's legal system, I would guess that the USA's is about middle of the spectrum in terms of badness/fairness/justice.
These things are hard to weigh objectively. For instance, in America the police don't take bribes, you can't bribe your way out of a traffic ticket. The cops will laugh at your attempt and pile on more charges. But if you're a local business owner, the bribes to local politicians are far from unheard of and all manner of corrupt dealings between business and local government is prevalent. So how you rank America's corruption depends on how you weigh those two forms of corruption. There's not one single objectively correct way to do that.
> For instance, in America the police don't take bribes, you can't bribe your way out of a traffic ticket. The cops will laugh at your attempt and pile on more charges.
Sure, they might not take as many bribes as South American police tends to take (as someone who traveled that continent in car without a driving license, I'd say 90% are accepting of bribes for minor crimes), but American police also accept bribes from time to time. They'll laugh at you and pile on more charges if you offer too little, but even American police has a price.
FY 2024 has 229 "Number of Bribery Offenses" (https://www.ussc.gov/research/quick-facts/bribery), which obviously doesn't account for the bribing that no one noticed or where there wasn't enough proof, we could probably assume it's at least 50% higher than that if we're being charitable, but in reality that number is probably way higher, by magnitudes.
I don't see anything about cops in that link. What I do see is that public officials were 49% of those charged but 45% were high-level elected officials. So that's what, maybe 4% that might be cops?
In Mexico, cops will pull people over just to collect chump change cash bribes. In America, you have people like Epstein bribing state attorney generals, but nobody even thinks to slip a cop a $100 bill with their drivers license. This sort of casual everyday roadside bribery does not exist in America.
> but nobody even thinks to slip a cop a $100 bill with their drivers license. This sort of casual everyday roadside bribery does not exist in America.
Obviously incorrect for both Southern and middle states in America. But sure, go on believing the US cops are somehow immune to corruption, which is something I never thought someone would honestly believe, even on the internet.
If you ever decide to visit America and learn what it's like firsthand, I encourage you to try this. Just make sure you know a good orthodontist first, and probably a good therapist too, because you're going to get thrown to the ground, handcuffed and sent to jail. There is no world in which trying to bribe your way out of an American traffic ticket makes more sense than just taking and paying the ticket. Even trying this is genuinely one of the stupidest things you could ever do.
Maybe it isn't the US government we need to worry about. What's stopping Flock from compiling and selling personal dossiers on every citizen like all the other big tech companies? They're just a private company so nothing to worry about, right?
