GitHub pip install s3explore
Data Engineering AI-Assisted Discovery

Before you build the pipeline,
understand the data.

Most teams skip data discovery — they jump to ETL and spend weeks fixing schemas, missing fields, and wrong assumptions. s3explore lets you query S3 files directly with SQL before a single pipeline is written.

$ pip install s3explore
View on GitHub →
150M
rows queried below
0
databases needed
5
CLI commands
<60s
to first insight

The Tool

SQL on S3. No infrastructure required.

s3explore wraps chDB (ClickHouse's embedded Python engine) and boto3. Point it at any S3 path and query Parquet, JSON, or CSV files directly — no Athena, no Glue, no cluster.

terminal
$ s3explore schema "s3://my-bucket/events/*.parquet" # → 15 columns: event_type, user_id, timestamp, region... $ s3explore count "s3://my-bucket/events/*.parquet" # → Row counts per file across all partitions $ s3explore query "s3://my-bucket/events/*.parquet" \   --sql "SELECT event_type, count() FROM {table} GROUP BY 1" # → Results in seconds. No database.
Parquet JSON / GZ CSV AWS SSO Public buckets

The AI Workflow

Give your AI an S3 bucket.
Get a data story back.

With --format json output, s3explore pipes cleanly into Claude Code. Drop the tool and its CLAUDE.md into any project — Claude reads all five commands and runs the full discovery sequence autonomously.

you
Engineer
"I have Parquet files landing in s3://my-bucket/events/year=2025/ — explore them and tell me what's interesting before I build the pipeline."
AI
Claude Code — running s3explore
ls schema count sample query
"Found 847 Parquet files across 12 months. Schema has 23 columns — event_type is 34% null. File sizes vary 10× between partitions suggesting uneven write loads. 8% of rows have user_id = null — worth flagging before ingestion."
💡
You just learned about a schema issue, a data quality problem, and a partition imbalance — before writing a single line of pipeline code. That's the point.

Live Example — Amazon Reviews

We ran it on a real public dataset.
Here's what came back.

Amazon's public reviews dataset lives in S3 as Parquet — no authentication needed. We queried it cold with five commands. 150 million rows. No database.

150M
total reviews
20 yrs
1995 → 2015
8
Parquet files
4.18★
overall avg rating
Review volume & avg rating by year
Volume exploded 220,000× from 1995 to 2014 — the mobile era changed everything
📉
Average ratings hit their lowest point in 2010–2011 (4.052★) as electronics and smartphone accessories flooded the platform — then recovered as verified purchases and content categories grew.

Finding 01

The J-curve: people review
when they love or hate.

62% of all reviews are 5-star. But 1-star outranks 2-star by 65% — the "mildly disappointed" almost never bother writing. Anger or delight. Nothing in between.

Star rating distribution — 150M reviews
The classic J-curve of online reviews
5★
61.7%
93.2M 61.7%
4★
17.4%
26.2M 17.4%
3★
8.0%
12.1M 8.0%
2★
4.8%
7.3M 4.8%
1★
8.0%
12.1M 8.0%

Finding 02

Tech categories are the
most frustrating to buy.

Books, Music, and Video score above 4.3★. Wireless, Home Entertainment, and Electronics can't crack 4.0★. Content satisfies. Hardware disappoints.

Avg star rating by category
High-volume categories only — ordered by rating
📱
Wireless (3.89★) is the worst-rated high-volume category — 9 million reviews, 20.7% of them negative. If this were your pipeline, you'd want to know that before building a star-rating feature.

Finding 03

Free product = harsher review.
Vine reviewers are more honest.

Vine reviewers get products for free — you'd expect gratitude. The data disagrees. They give fewer 5-stars and their reviews get 2× more helpful votes. Obligation to be thorough beats incentive to be positive.

Regular Reviewer
Avg rating 4.20★
% 5-star 61.8%
Avg helpful votes 1.9
Total reviews 150.3M
Vine Reviewer — free product
Avg rating 4.08★
% 5-star 40.4%
Avg helpful votes 3.76
Total reviews 666K
Verified Purchase
Avg rating 4.23★
Avg helpful votes 1.31
Unverified Purchase
Avg rating 4.10★
Avg helpful votes 4.00

Try It Now

No AWS account needed.
Run it in two minutes.

The Amazon reviews dataset is public. Install s3explore and run the query below — it works out of the box. No credentials, no setup.

no AWS account needed
$ pip install s3explore $ s3explore query \   "s3://datasets-documentation/amazon_reviews/*.parquet" \   --sql "SELECT product_category,          round(avg(star_rating), 2) AS avg_stars,          count() AS reviews          FROM {table}          GROUP BY 1 ORDER BY 2 DESC" # → 150M rows. Results in seconds. No database.