Just to echo the OP -- the Cloud SQL proxy has been a pain point for me for years, it makes deploying to Kubernetes a PITA, since I have to use a different configuration vs. my dev setup. Whitelisting RDS networks inside the VPS is a feature that's existed in AWS for many many years.
Also agree that Stackdriver is pretty underwhelming, though it does have the perk of providing zero-configuration log capture in GKE.
Lots of problems though -
* UI is super-slow, yet it drops old records when you scroll, so scrolling up and down by more than 100 lines causes a reload of the logs I was just looking at. (Note that e.g. Kibana will do infinite scrolling to load more log entries, but doesn't drop old ones when you scroll, which is much more usable).
* It's really hard to work with large log traces; I'd love a quick way to "show raw logs for this time-period/search" so that I can either copy into a text editor or use browser search to explore. An example use-case here is "select all logs with this request correlator", which could include thousands of logs from a request, then jump around that log stream to look for interesting events. Re-running the search is painfully slow for exploring results like this.
* I've also had problems with alerts; the algorithm used for uptime checks is pretty naive, it just checks if the endpoint was down for the entire threshold period, which doesn't catch a flapping endpoint (e.g. down for 1s, up for 1s). Grafana can be configured to detect that the average latency spiked over the period, even if it's not fully down. This has led to us not being alerted for a production outage, so I've moved our critical alerting off Stackdriver. (Again, the promise here is great; if you could get alerting working as well as Grafana, it would be super-powerful to be able to alert on any synthetic metric generated from any log stream in the cluster. It's just half-baked right now.)
All that said, I've been happy with GCP in general, in particular GKE has made running a k8s cluster a lot easier.
I'm trying to reproduce the Cloud SQL Proxy in Cloud Shell that both you and the OP have mentioned, but I've been unable to reproduce (and thus file a bug). I'm sure I'm missing a step. If you're willing, I would love if you could drop me a quick note (sethvargo at google dot com) with the steps you're taking, commands you're running, and output you're seeing. This will help me escalate it to the right team.
I'm also going to share your feedback about Stackdriver with our product team as well. Thank you for that!
Just to clarify with some more detail (for the HN record), the issue with Cloud SQL is that while you can whitelist an individual internal IP, you can't whitelist an internal network (like 10/8). This means you need a sidecar service of some sort to add the whitelist rules when a new instance/pod comes up that should be able to access the DB.
In AWS for a simple app I'd just whitelist 10/8 and have the DB open to all internal instances (with TLS for added security as required).
I'll follow up by email too, happy to provide detailed feedback on this sort of thing.
Thanks for taking the time to engage with the community on this stuff!
We have similar experience with StackDriver Logging. It's felt unfinished since day one. Two problems come to mind:
(1) Some time ago I needed to get a month of logs (filtered by an expression) in order to compare the logged data against a database. The Console does not have any such option. You can set up "exporting", but that only starts replication from the moment you set it up; no historical data. You can use the APIs or "gcloud beta logging read", but the performance is terrible. I got maybe 200-500KB/sec. After keeping the command running continuously for 3 days (!), I still only had 20% of the log data.
These days, we have exporting set up to pipe all the logs into BigQuery and GCS, just to get around the crappiness. That gives some query capabilities, as well as the ability to quickly grab the original data for processing with other tools.
(2) We're on GKE, and none of the container's labels end up in log entries. Surely this is pretty key stuff. There seems to be no way to customize the payloads that GKE's Fluentd sends to StackDriver.
(3) There's no way to tail logs. The gcloud command doesn't have it, and the Console doesn't either. It feels like such a missed opportunity.
I think Google would be better off ditching StackDriver Logging entirely and instead let the user set up rules to send logs to different services that actually work. The default could be BigQuery. I don't understand the point of having a separate, proprietary query language for logs when you have SQL.
I've looked briefly at the other StackDriver stuff (most of which is still on stackdriver.com, always causing a confusing redirect and often re-authentication), and it's been similarly underwhelming. We use Prometheus with Grafana, and StackDriver seems pointless in comparison.
the biggest problem I had with cloud sql proxy or cloud sql in generell is that postgresql is limited to 100 connections.
since I'm a small customer hosting many low traffic projects I want to share my instance as much as possible, however running pgbouncer for a lot of ups is painful since cloudsql does not allow to query the auth table (what's used by pgbouncer if you don't add all database username/password on your own).
I think a lot of problems would be solved if there would be a way to setup cloud sql proxy that comes with transaction pooling, especially when using k8s and having a pooling namespace (multiple load balancers for HA) where all other apps connect to.
also cloud sql and ipv6 is not funny. (i.e. a lot of things in gcp is ipv4 only which is sad)
Thank you, this is very helpful because these little things are the stuff that you can’t find in any documentation but end up biting you in production.
Also agree that Stackdriver is pretty underwhelming, though it does have the perk of providing zero-configuration log capture in GKE.
Lots of problems though -
* UI is super-slow, yet it drops old records when you scroll, so scrolling up and down by more than 100 lines causes a reload of the logs I was just looking at. (Note that e.g. Kibana will do infinite scrolling to load more log entries, but doesn't drop old ones when you scroll, which is much more usable).
* It's really hard to work with large log traces; I'd love a quick way to "show raw logs for this time-period/search" so that I can either copy into a text editor or use browser search to explore. An example use-case here is "select all logs with this request correlator", which could include thousands of logs from a request, then jump around that log stream to look for interesting events. Re-running the search is painfully slow for exploring results like this.
* I've also had problems with alerts; the algorithm used for uptime checks is pretty naive, it just checks if the endpoint was down for the entire threshold period, which doesn't catch a flapping endpoint (e.g. down for 1s, up for 1s). Grafana can be configured to detect that the average latency spiked over the period, even if it's not fully down. This has led to us not being alerted for a production outage, so I've moved our critical alerting off Stackdriver. (Again, the promise here is great; if you could get alerting working as well as Grafana, it would be super-powerful to be able to alert on any synthetic metric generated from any log stream in the cluster. It's just half-baked right now.)
All that said, I've been happy with GCP in general, in particular GKE has made running a k8s cluster a lot easier.