Compare commits

...

3 Commits

Author SHA1 Message Date
Deeman
8f2ffd432b fix(admin): correct docker volume mount + pipeline_routes repo root
All checks were successful
CI / test (push) Successful in 50s
CI / tag (push) Successful in 2s
- docker-compose.prod.yml: fix volume mount for all 6 web containers
  from /opt/padelnomics/data (stale) → /data/padelnomics (live supervisor output);
  add LANDING_DIR=/app/data/pipeline/landing so extraction/landing stats work
- pipeline_routes.py: fix _REPO_ROOT parents[5] → parents[4] so workflows.toml
  is found in dev and pipeline overview shows workflow schedules

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-01 11:41:29 +01:00
Deeman
c9dec066f7 fix(admin): mobile UX fixes — contrast, scroll, responsive grids
- CSS: `.nav-mobile a` → `.nav-mobile a:not(.nav-auth-btn)` to fix Sign
  Out button showing slate text instead of white on mobile
- base_admin.html: add `overflow-y: hidden` + `scrollbar-width: none` to
  `.admin-subnav` to eliminate ghost 1px scrollbar on Content tab row
- routes.py: pass `outreach_email=EMAIL_ADDRESSES["outreach"]` to outreach
  template so sending domain is no longer hardcoded
- outreach.html: display dynamic `outreach_email`; replace inline
  `repeat(6,1fr)` grid with responsive `.pipeline-status-grid` (2→3→6 cols)
- index.html: replace inline `repeat(5,1fr)` Lead/Supplier Funnel grids
  with responsive `.funnel-grid` class (2 cols mobile, 5 cols md+)
- pipeline.html: replace inline `repeat(4,1fr)` stat grid with responsive
  `.pipeline-stat-grid` (2 cols mobile, 4 cols md+)
- 4 partials (lead/email/supplier/outreach results): wrap `<table>` in
  `<div style="overflow-x:auto">` so tables scroll on narrow screens

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-01 11:20:46 +01:00
Deeman
fea4f85da3 perf(transform): optimize dim_locations spatial joins via IEJoin + country filters
All checks were successful
CI / test (push) Successful in 51s
CI / tag (push) Successful in 2s
Replace ABS() bbox predicates with BETWEEN in all three spatial CTEs
(nearest_padel, padel_local, tennis_nearby). BETWEEN enables DuckDB's
IEJoin (interval join) which is O((N+M) log M) vs the previous O(N×M)
nested-loop cross-join.

Add country pre-filters to restrict the left side from ~140K global
locations to ~20K rows for padel/tennis CTEs (~8 countries each).

Expected: ~50-200x speedup on the spatial CTE portion of the model.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-01 02:57:05 +01:00
14 changed files with 106 additions and 29 deletions

View File

@@ -17,6 +17,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- 15 new tests in `web/tests/test_affiliate.py` (41 total) - 15 new tests in `web/tests/test_affiliate.py` (41 total)
### Fixed ### Fixed
- **Data Platform admin view showing stale/zero row counts** — Docker web containers were mounting `/opt/padelnomics/data` (stale copy) instead of `/data/padelnomics` (live supervisor output). Fixed volume mount in all 6 containers (blue/green × app/worker/scheduler) and added `LANDING_DIR=/app/data/pipeline/landing` so extraction stats and landing zone file stats are visible to the web app.
- **`workflows.toml` never found in dev** — `_REPO_ROOT` in `pipeline_routes.py` used `parents[5]` (one level too far up) instead of `parents[4]`. Workflow schedules now display correctly on the pipeline overview tab in dev.
- **Article preview frontmatter bug** — `_rebuild_article()` in `admin/routes.py` now strips YAML frontmatter before passing markdown to `mistune.html()`, preventing raw `title:`, `slug:` etc. from appearing as visible text in article previews. - **Article preview frontmatter bug** — `_rebuild_article()` in `admin/routes.py` now strips YAML frontmatter before passing markdown to `mistune.html()`, preventing raw `title:`, `slug:` etc. from appearing as visible text in article previews.
### Added ### Added

View File

@@ -60,9 +60,10 @@ services:
environment: environment:
- DATABASE_PATH=/app/data/app.db - DATABASE_PATH=/app/data/app.db
- SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb - SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb
- LANDING_DIR=/app/data/pipeline/landing
volumes: volumes:
- app-data:/app/data - app-data:/app/data
- /opt/padelnomics/data:/app/data/pipeline:ro - /data/padelnomics:/app/data/pipeline:ro
networks: networks:
- net - net
healthcheck: healthcheck:
@@ -82,9 +83,10 @@ services:
environment: environment:
- DATABASE_PATH=/app/data/app.db - DATABASE_PATH=/app/data/app.db
- SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb - SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb
- LANDING_DIR=/app/data/pipeline/landing
volumes: volumes:
- app-data:/app/data - app-data:/app/data
- /opt/padelnomics/data:/app/data/pipeline:ro - /data/padelnomics:/app/data/pipeline:ro
networks: networks:
- net - net
@@ -98,9 +100,10 @@ services:
environment: environment:
- DATABASE_PATH=/app/data/app.db - DATABASE_PATH=/app/data/app.db
- SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb - SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb
- LANDING_DIR=/app/data/pipeline/landing
volumes: volumes:
- app-data:/app/data - app-data:/app/data
- /opt/padelnomics/data:/app/data/pipeline:ro - /data/padelnomics:/app/data/pipeline:ro
networks: networks:
- net - net
@@ -115,9 +118,10 @@ services:
environment: environment:
- DATABASE_PATH=/app/data/app.db - DATABASE_PATH=/app/data/app.db
- SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb - SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb
- LANDING_DIR=/app/data/pipeline/landing
volumes: volumes:
- app-data:/app/data - app-data:/app/data
- /opt/padelnomics/data:/app/data/pipeline:ro - /data/padelnomics:/app/data/pipeline:ro
networks: networks:
- net - net
healthcheck: healthcheck:
@@ -137,9 +141,10 @@ services:
environment: environment:
- DATABASE_PATH=/app/data/app.db - DATABASE_PATH=/app/data/app.db
- SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb - SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb
- LANDING_DIR=/app/data/pipeline/landing
volumes: volumes:
- app-data:/app/data - app-data:/app/data
- /opt/padelnomics/data:/app/data/pipeline:ro - /data/padelnomics:/app/data/pipeline:ro
networks: networks:
- net - net
@@ -153,9 +158,10 @@ services:
environment: environment:
- DATABASE_PATH=/app/data/app.db - DATABASE_PATH=/app/data/app.db
- SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb - SERVING_DUCKDB_PATH=/app/data/pipeline/analytics.duckdb
- LANDING_DIR=/app/data/pipeline/landing
volumes: volumes:
- app-data:/app/data - app-data:/app/data
- /opt/padelnomics/data:/app/data/pipeline:ro - /data/padelnomics:/app/data/pipeline:ro
networks: networks:
- net - net

View File

@@ -19,8 +19,10 @@
-- 4. Country-level income (global fallback from stg_income / ilc_di03) -- 4. Country-level income (global fallback from stg_income / ilc_di03)
-- --
-- Distance calculations use ST_Distance_Sphere (DuckDB spatial extension). -- Distance calculations use ST_Distance_Sphere (DuckDB spatial extension).
-- A bounding-box pre-filter (~0.5°, ≈55km) reduces the cross-join before the -- Spatial joins use BETWEEN predicates (not ABS()) to enable DuckDB's IEJoin
-- exact sphere distance is computed. -- (interval join) optimization: O((N+M) log M) vs O(N×M) nested-loop.
-- Country pre-filters restrict the left side to ~20K rows for padel/tennis CTEs
-- (~8 countries each), down from ~140K global locations.
MODEL ( MODEL (
name foundation.dim_locations, name foundation.dim_locations,
@@ -147,6 +149,8 @@ padel_courts AS (
WHERE lat IS NOT NULL AND lon IS NOT NULL WHERE lat IS NOT NULL AND lon IS NOT NULL
), ),
-- Nearest padel court distance per location (bbox pre-filter → exact sphere distance) -- Nearest padel court distance per location (bbox pre-filter → exact sphere distance)
-- BETWEEN enables DuckDB IEJoin (O((N+M) log M)) vs ABS() nested-loop (O(N×M)).
-- Country pre-filter reduces left side from ~140K to ~20K rows (padel is ~8 countries).
nearest_padel AS ( nearest_padel AS (
SELECT SELECT
l.geoname_id, l.geoname_id,
@@ -158,9 +162,12 @@ nearest_padel AS (
) AS nearest_padel_court_km ) AS nearest_padel_court_km
FROM locations l FROM locations l
JOIN padel_courts p JOIN padel_courts p
-- ~55km bounding box pre-filter to limit cross-join before sphere calc -- ~55km bounding box pre-filter; BETWEEN triggers IEJoin optimization
ON ABS(l.lat - p.lat) < 0.5 ON l.lat BETWEEN p.lat - 0.5 AND p.lat + 0.5
AND ABS(l.lon - p.lon) < 0.5 AND l.lon BETWEEN p.lon - 0.5 AND p.lon + 0.5
WHERE l.country_code IN (
SELECT DISTINCT country_code FROM padel_courts WHERE country_code IS NOT NULL
)
GROUP BY l.geoname_id GROUP BY l.geoname_id
), ),
-- Padel venues within 5km of each location (counts as "local padel supply") -- Padel venues within 5km of each location (counts as "local padel supply")
@@ -170,24 +177,35 @@ padel_local AS (
COUNT(*) AS padel_venue_count COUNT(*) AS padel_venue_count
FROM locations l FROM locations l
JOIN padel_courts p JOIN padel_courts p
ON ABS(l.lat - p.lat) < 0.05 -- ~5km bbox pre-filter -- ~5km bbox pre-filter; BETWEEN triggers IEJoin optimization
AND ABS(l.lon - p.lon) < 0.05 ON l.lat BETWEEN p.lat - 0.05 AND p.lat + 0.05
WHERE ST_Distance_Sphere( AND l.lon BETWEEN p.lon - 0.05 AND p.lon + 0.05
WHERE l.country_code IN (
SELECT DISTINCT country_code FROM padel_courts WHERE country_code IS NOT NULL
)
AND ST_Distance_Sphere(
ST_Point(l.lon, l.lat), ST_Point(l.lon, l.lat),
ST_Point(p.lon, p.lat) ST_Point(p.lon, p.lat)
) / 1000.0 <= 5.0 ) / 1000.0 <= 5.0
GROUP BY l.geoname_id GROUP BY l.geoname_id
), ),
-- Tennis courts within 25km of each location (sports culture proxy) -- Tennis courts within 25km of each location (sports culture proxy)
-- Country pre-filter reduces left side from ~140K to ~20K rows (tennis courts are European only).
tennis_nearby AS ( tennis_nearby AS (
SELECT SELECT
l.geoname_id, l.geoname_id,
COUNT(*) AS tennis_courts_within_25km COUNT(*) AS tennis_courts_within_25km
FROM locations l FROM locations l
JOIN staging.stg_tennis_courts t JOIN staging.stg_tennis_courts t
ON ABS(l.lat - t.lat) < 0.23 -- ~25km bbox pre-filter -- ~25km bbox pre-filter; BETWEEN triggers IEJoin optimization
AND ABS(l.lon - t.lon) < 0.23 ON l.lat BETWEEN t.lat - 0.23 AND t.lat + 0.23
WHERE ST_Distance_Sphere( AND l.lon BETWEEN t.lon - 0.23 AND t.lon + 0.23
WHERE l.country_code IN (
SELECT DISTINCT country_code
FROM staging.stg_tennis_courts
WHERE country_code IS NOT NULL
)
AND ST_Distance_Sphere(
ST_Point(l.lon, l.lat), ST_Point(l.lon, l.lat),
ST_Point(t.lon, t.lat) ST_Point(t.lon, t.lat)
) / 1000.0 <= 25.0 ) / 1000.0 <= 25.0

View File

@@ -49,7 +49,7 @@ _LANDING_DIR = os.environ.get("LANDING_DIR", "data/landing")
_SERVING_DUCKDB_PATH = os.environ.get("SERVING_DUCKDB_PATH", "data/analytics.duckdb") _SERVING_DUCKDB_PATH = os.environ.get("SERVING_DUCKDB_PATH", "data/analytics.duckdb")
# Repo root: web/src/padelnomics/admin/ → up 4 levels # Repo root: web/src/padelnomics/admin/ → up 4 levels
_REPO_ROOT = Path(__file__).resolve().parents[5] _REPO_ROOT = Path(__file__).resolve().parents[4]
_WORKFLOWS_TOML = _REPO_ROOT / "infra" / "supervisor" / "workflows.toml" _WORKFLOWS_TOML = _REPO_ROOT / "infra" / "supervisor" / "workflows.toml"
# A "running" row older than this is considered stale/crashed. # A "running" row older than this is considered stale/crashed.

View File

@@ -3037,6 +3037,7 @@ async def outreach():
current_search=search, current_search=search,
current_follow_up=follow_up, current_follow_up=follow_up,
page=page, page=page,
outreach_email=EMAIL_ADDRESSES["outreach"],
) )

View File

@@ -40,8 +40,10 @@
.admin-subnav { .admin-subnav {
display: flex; align-items: stretch; padding: 0 2rem; display: flex; align-items: stretch; padding: 0 2rem;
background: #fff; border-bottom: 1px solid #E2E8F0; background: #fff; border-bottom: 1px solid #E2E8F0;
flex-shrink: 0; overflow-x: auto; gap: 0; flex-shrink: 0; overflow-x: auto; overflow-y: hidden; gap: 0;
scrollbar-width: none;
} }
.admin-subnav::-webkit-scrollbar { display: none; }
.admin-subnav a { .admin-subnav a {
display: flex; align-items: center; gap: 5px; display: flex; align-items: center; gap: 5px;
padding: 0 1px; margin: 0 13px 0 0; height: 42px; padding: 0 1px; margin: 0 13px 0 0; height: 42px;

View File

@@ -3,6 +3,19 @@
{% block title %}Admin Dashboard - {{ config.APP_NAME }}{% endblock %} {% block title %}Admin Dashboard - {{ config.APP_NAME }}{% endblock %}
{% block admin_head %}
<style>
.funnel-grid {
display: grid;
grid-template-columns: repeat(2, 1fr);
gap: 0.75rem;
}
@media (min-width: 768px) {
.funnel-grid { grid-template-columns: repeat(5, 1fr); }
}
</style>
{% endblock %}
{% block admin_content %} {% block admin_content %}
<header class="flex justify-between items-center mb-8"> <header class="flex justify-between items-center mb-8">
<div> <div>
@@ -47,7 +60,7 @@
<!-- Lead Funnel --> <!-- Lead Funnel -->
<p class="text-xs font-semibold text-slate uppercase tracking-wider mb-2">Lead Funnel</p> <p class="text-xs font-semibold text-slate uppercase tracking-wider mb-2">Lead Funnel</p>
<div style="display:grid;grid-template-columns:repeat(5,1fr);gap:0.75rem" class="mb-8"> <div class="funnel-grid mb-8">
<div class="card text-center border-l-4 border-l-electric" style="padding:0.75rem"> <div class="card text-center border-l-4 border-l-electric" style="padding:0.75rem">
<p class="text-xs text-slate">Planner Users</p> <p class="text-xs text-slate">Planner Users</p>
<p class="text-xl font-bold text-navy">{{ stats.planner_users }}</p> <p class="text-xl font-bold text-navy">{{ stats.planner_users }}</p>
@@ -72,7 +85,7 @@
<!-- Supplier Stats --> <!-- Supplier Stats -->
<p class="text-xs font-semibold text-slate uppercase tracking-wider mb-2">Supplier Funnel</p> <p class="text-xs font-semibold text-slate uppercase tracking-wider mb-2">Supplier Funnel</p>
<div style="display:grid;grid-template-columns:repeat(5,1fr);gap:0.75rem" class="mb-8"> <div class="funnel-grid mb-8">
<div class="card text-center border-l-4 border-l-accent" style="padding:0.75rem"> <div class="card text-center border-l-4 border-l-accent" style="padding:0.75rem">
<p class="text-xs text-slate">Claimed Suppliers</p> <p class="text-xs text-slate">Claimed Suppliers</p>
<p class="text-xl font-bold text-navy">{{ stats.suppliers_claimed }}</p> <p class="text-xl font-bold text-navy">{{ stats.suppliers_claimed }}</p>

View File

@@ -2,13 +2,30 @@
{% set admin_page = "outreach" %} {% set admin_page = "outreach" %}
{% block title %}Outreach Pipeline - Admin - {{ config.APP_NAME }}{% endblock %} {% block title %}Outreach Pipeline - Admin - {{ config.APP_NAME }}{% endblock %}
{% block admin_head %}
<style>
.pipeline-status-grid {
display: grid;
grid-template-columns: repeat(2, 1fr);
gap: 0.75rem;
margin-bottom: 1.5rem;
}
@media (min-width: 640px) {
.pipeline-status-grid { grid-template-columns: repeat(3, 1fr); }
}
@media (min-width: 1024px) {
.pipeline-status-grid { grid-template-columns: repeat(6, 1fr); }
}
</style>
{% endblock %}
{% block admin_content %} {% block admin_content %}
<header class="flex justify-between items-center mb-6"> <header class="flex justify-between items-center mb-6">
<div> <div>
<h1 class="text-2xl">Outreach</h1> <h1 class="text-2xl">Outreach</h1>
<p class="text-sm text-slate mt-1"> <p class="text-sm text-slate mt-1">
{{ pipeline.total }} supplier{{ 's' if pipeline.total != 1 else '' }} in pipeline {{ pipeline.total }} supplier{{ 's' if pipeline.total != 1 else '' }} in pipeline
&middot; Sending domain: <span class="mono text-xs">hello.padelnomics.io</span> &middot; Sending from: <span class="mono text-xs">{{ outreach_email }}</span>
</p> </p>
</div> </div>
<div class="flex gap-2"> <div class="flex gap-2">
@@ -18,7 +35,7 @@
</header> </header>
<!-- Pipeline cards --> <!-- Pipeline cards -->
<div style="display:grid;grid-template-columns:repeat(6,1fr);gap:0.75rem;margin-bottom:1.5rem"> <div class="pipeline-status-grid">
{% set status_colors = { {% set status_colors = {
'prospect': '#E2E8F0', 'prospect': '#E2E8F0',
'contacted': '#DBEAFE', 'contacted': '#DBEAFE',

View File

@@ -1,5 +1,6 @@
{% if emails %} {% if emails %}
<div class="card"> <div class="card">
<div style="overflow-x:auto">
<table class="table"> <table class="table">
<thead> <thead>
<tr> <tr>
@@ -39,6 +40,7 @@
</tbody> </tbody>
</table> </table>
</div> </div>
</div>
{% else %} {% else %}
<div class="card text-center" style="padding:2rem"> <div class="card text-center" style="padding:2rem">
<p class="text-slate">No emails match the current filters.</p> <p class="text-slate">No emails match the current filters.</p>

View File

@@ -25,6 +25,7 @@
{% if leads %} {% if leads %}
<div class="card"> <div class="card">
<div style="overflow-x:auto">
<table class="table"> <table class="table">
<thead> <thead>
<tr> <tr>
@@ -59,6 +60,7 @@
</tbody> </tbody>
</table> </table>
</div> </div>
</div>
<!-- Pagination --> <!-- Pagination -->
{% if total > per_page %} {% if total > per_page %}

View File

@@ -1,5 +1,6 @@
{% if suppliers %} {% if suppliers %}
<div class="card"> <div class="card">
<div style="overflow-x:auto">
<table class="table"> <table class="table">
<thead> <thead>
<tr> <tr>
@@ -20,6 +21,7 @@
</tbody> </tbody>
</table> </table>
</div> </div>
</div>
{% else %} {% else %}
<div class="card text-center" style="padding:2rem"> <div class="card text-center" style="padding:2rem">
<p class="text-slate">No suppliers match the current filters.</p> <p class="text-slate">No suppliers match the current filters.</p>

View File

@@ -1,5 +1,6 @@
{% if suppliers %} {% if suppliers %}
<div class="card"> <div class="card">
<div style="overflow-x:auto">
<table class="table"> <table class="table">
<thead> <thead>
<tr> <tr>
@@ -48,6 +49,7 @@
</tbody> </tbody>
</table> </table>
</div> </div>
</div>
{% else %} {% else %}
<div class="card text-center" style="padding:2rem"> <div class="card text-center" style="padding:2rem">
<p class="text-slate">No suppliers match the current filters.</p> <p class="text-slate">No suppliers match the current filters.</p>

View File

@@ -4,6 +4,15 @@
{% block admin_head %} {% block admin_head %}
<style> <style>
.pipeline-stat-grid {
display: grid;
grid-template-columns: repeat(2, 1fr);
gap: 0.75rem;
}
@media (min-width: 768px) {
.pipeline-stat-grid { grid-template-columns: repeat(4, 1fr); }
}
.pipeline-tabs { .pipeline-tabs {
display: flex; gap: 0; border-bottom: 2px solid #E2E8F0; margin-bottom: 1.5rem; display: flex; gap: 0; border-bottom: 2px solid #E2E8F0; margin-bottom: 1.5rem;
} }
@@ -46,7 +55,7 @@
</header> </header>
<!-- Health stat cards --> <!-- Health stat cards -->
<div style="display:grid;grid-template-columns:repeat(4,1fr);gap:0.75rem" class="mb-6"> <div class="pipeline-stat-grid mb-6">
<div class="card text-center" style="padding:0.875rem"> <div class="card text-center" style="padding:0.875rem">
<p class="text-xs text-slate">Total Runs</p> <p class="text-xs text-slate">Total Runs</p>
<p class="text-2xl font-bold text-navy metric">{{ summary.total | default(0) }}</p> <p class="text-2xl font-bold text-navy metric">{{ summary.total | default(0) }}</p>

View File

@@ -218,9 +218,7 @@
.nav-bar[data-navopen="true"] .nav-mobile { .nav-bar[data-navopen="true"] .nav-mobile {
display: flex; display: flex;
} }
.nav-mobile a, .nav-mobile a:not(.nav-auth-btn) {
.nav-mobile button.nav-auth-btn,
.nav-mobile a.nav-auth-btn {
display: block; display: block;
padding: 0.625rem 0; padding: 0.625rem 0;
border-bottom: 1px solid #F1F5F9; border-bottom: 1px solid #F1F5F9;
@@ -230,15 +228,18 @@
text-decoration: none; text-decoration: none;
transition: color 0.15s; transition: color 0.15s;
} }
.nav-mobile a:last-child { border-bottom: none; } .nav-mobile a:not(.nav-auth-btn):last-child { border-bottom: none; }
.nav-mobile a:hover { color: #1D4ED8; } .nav-mobile a:not(.nav-auth-btn):hover { color: #1D4ED8; }
/* nav-auth-btn in mobile menu: override block style, keep button colours */
.nav-mobile a.nav-auth-btn, .nav-mobile a.nav-auth-btn,
.nav-mobile button.nav-auth-btn { .nav-mobile button.nav-auth-btn {
display: inline-flex; display: inline-flex;
margin-top: 0.5rem; margin-top: 0.5rem;
padding: 6px 16px;
border-bottom: none; border-bottom: none;
width: auto; width: auto;
align-self: flex-start; align-self: flex-start;
color: #fff;
} }
.nav-mobile .nav-mobile-section { .nav-mobile .nav-mobile-section {
font-size: 0.6875rem; font-size: 0.6875rem;